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(b) All the claims are believed to be directed to a single invention. If the 
Office determines that all the claims presented are not obviously directed to a single 
invention, then Applicants will make an election without traverse as a prerequisite to the 
grant of special status. 

(c) Pre-examination searches were made of U.S. issued patents, including 
a classification search, a computer database search, and a keyword search. The searches were 
performed on or aromid January 14, 2005, and were conducted by a professional search firm, 
Kramer & Amado, P.C. The classification search covered Class 711 (subclasses 162, 207, 
and 209) for the U.S. and foreign subclasses identified above. The computer database search 
was conducted on the USPTO systems EAST aad WEST. The keyword search was 
conducted in Class 711 (subclasses 111, 148, 151, and 159). The inventors fiirther provided a 
reference considered most closely related to the subject matter of the present apphcation (see 
reference #4 below), which was cited in the Information Disclosure Statements filed on 
March 18, 2004. 

(d) The following references, copies of which are attached herewith, are 
deemed most closely related to the subject matter encompassed by the claims: 

(1) U.S. Patent No. 4,843,543; 

(2) U.S. Patent No. 6,529,944 B 1 ; 

(3) U.S. Patent No. 4,985,828; and 

(4) Japanese Patent PubUcation No. JP 2002- 1 32552. 

(e) Set forth below is a detailed discussion of references which points out 
with particularity how the claimed subject matter is distinguishable over the references. 

A. Claimed Embodiments of the Present Invention 

The claimed embodiments relate to a storage device management method in a 
computer system, a computer system, a storage device, and a management computer, 
particularly, to a method for managing a storage device in a computer system capable of 
effectively grasping the storage device use state and the file allocation information. 
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Independent claim 1 recites a storage management method for a computer 
system including a host computer, a storage sub-system having one or more volumes, a disk 
control unit provided in the storage sub-system for controlling operation of the storage sub- 
system, and a management computer for managing configuration of the storage sub-system. 
The method comprises registering, by the host computer, a volxune identifier, a physical 
address, and a group identifier of the storage sub-system in a disk management table in the 
disk control units as a result of grouping of volumes in the storage sub-system according to a 
use purpose; obtaining, by the disk control unit, a physical address of a volvmie belonging to a 
same group as a group identifier input fi-om the management computer fi-om the disk 
management table, and notifying information described in a volume list of a volume allocated 
at the physical address to the management computer; and displaying, by the management 
computer, the information thus notified. 

Independent claim 2 recites a storage management method for a computer 
system including a host computer, a storage sub-system having one or more volumes, a disk 
control unit provided in the storage sub-system for controlling operation of the storage sub- 
system, and a management computer for managing configuration of the storage sub-system. 
The method comprises registering, by the host computer, a volume identifier, a physical 
address, and a group identifier of the storage sub-system in the disk management table in a 
disk control imit as a result of grouping of volimies in the storage sub-system according to a 
use purpose; storing, by the disk control unit, information on the volume identifier and the 
physical address of the storage sub-system acquired at system starting in the disk 
management table, obtaining fi-om the disk management table a physical address of a volume 
belonging to a same group as a group input fi-om the management computer, and notifying 
information described in a volume list of a volume allocated at the physical address to the 
management computer; and displaying, by the management computer, the information thus 
notified. 

Independent claim 5 recites a computer system comprising a host computer; a 
storage sub-system having one or more volumes and a disk control imit; and a management 
computer configured to manage configuration of the storage sub-system. The host computer 
transmits a volume identifier, a physical address, and a group identifier of the storage sub- 
system as a result of grouping of volimies in a storage sub-system according to a use purpose. 
The disk control unit obtains a physical address of a volxmie belonging to a same group as a 
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group transmitted from the management computer, and notifies information described in a 
volvmie list of a volume allocated at the physical address to the management computer. The 
management computer displays the information thus notified. 

Independent claim 8 recites a storage system comprising a plurality of storage 
devices having one or more volumes; and a disk control unit connected to a host computer 
and a management computer configured to control the plurality of storage devices. The disk 
control unit further includes a disk management table for storing a volume identifier, a 
physical address, and a group identifier of a storage device as a result of grouping of volumes 
in the storage device according to a use purpose. The disk control imit obtains a physical 
address of a volume belonging to a same group as a group transmitted from the management 
computer, and notifies information described in a volume list of a volxmie allocated at the 
physical address to the management computer. 

Lidependent claim 9 recites a management computer comprising a control 
section; and a storage device. The control section comprises a plurality of pairs of one or 
more voliunes and a disk control unit for managing configuration of storage used by a host 
computer. Information described in a disk management table in a disk control unit 
transmitted from disk control units is stored together with identifiers of the disk control imits 
in the storage device, a disk control unit to which a volume corresponding to a requested 
group or a requested volume identifier belongs is requested to acquire information on a 
volimie list concerning the volume belonging to the group or voliraie thus requested, and 
according to information on a volume list as a response to this request, an information file of 
the disk management table stored in the storage device is updated to the disk management 
table information thus notified. 

Independent claim 10 recites a management program stored in a storage 
mediimi and executed in a management computer including a plurality of pairs of one or 
more volumes and a disk control device for managing configuration of a storage sub-system 
used by a host computer. The program comprises code for storing information described on a 
disk management table in a disk control imit from respective disk control units together with 
identifiers of the disk control units into a storage device accessible by the management 
computer; code for requesting the disk control unit to which a volume corresponding to a 
requested group or a requested volume identifier belongs, to acquire volume list information 
conceming a volxmie belonging to the group or the volume identifier thus requested; and code 
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for, according to the volume list information as a response to the request, updating an 
information file in the disk management table stored in the storage device accessible by the 
management computer to the disk management table information thus notified. 

Independent claim 1 1 recites a computer system comprising a host computer; 
a storage sub-system having one or more volumes and a disk control unit; and a management 
computer configured to manage configuration of the storage sub-system. The host computer 
transmits a volume identifier, a physical address, and a group identifier of the storage sub- 
system as a result of grouping of volimies in the storage sub-system according to a use 
purpose. The disk control unit obtains a physical address of a volume belonging to a same 
group as a group transmitted fi-om the management computer, and notifies information 
described in a volume list of a volume allocated at the physical address to the management 
computer. The management computer displays the information thus notified. The 
management computer stores information described in the disk management table together 
with a disk control unit identifier fi-om respective disk control units to a storage device 
accessible by the management computer, and requests a disk control imit to which a volume 
corresponding to a requested group or a requested identifier belongs to acquire volvime list 
information concerning the volume belonging to the group or the volume identifier thus 
requested. The disk control unit which has received the request notifies information 
described in a volume list of a volume identical to the group or the volimie identifier 
specified by the management computer to the management computer. Each of the control 
units notifies information requested to be acquired fi-om the management computer to the 
management computer, and notifies new information on the disk management table to the 
management computer. The management table thus notified updates an information file of 
the disk management table stored in the storage device accessible by the management 
computer to the disk management table thus notified. 

One of the benefits that may be derived is that it is possible to effectively 
acquire the storage device information such as information on the storage device use 
condition and information on the files stored without increasing the load on the host 
computer, and it is possible to acquire information for managing the large-scale storage 
system use condition without increasing the load on the host computer. 
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B. Discussion of the References 

1. U.S. Patent No. 4,843,543 

This reference relates to a storage control method and apparatus with a 
plurality of identifier transmission means for dividing the access requests issued firom the 
plurahty of access request control vmits into a plurality of groups based on the order of 
issuance from the access request control units. Each identifier transmission means is 
associated with one of the access request xmits and divides the access requests from the 
associated access request imit. Each identifier transmission means fiirther adds access 
requests in each group with access request identifiers. 

The reference does not teach registering, by the host computer, a volume 
identifier, a physical address, and a group identifier of the storage sub-system in a disk 
management table in the disk control units as a result of grouping of volumes in the storage 
sub-system according to a use purpose; obtaining, by the disk control unit, a physical address 
of a volume belonging to a same group as a group identifier input from the management 
computer from the disk management table, and notifying information described in a volume 
list of a volume allocated at the physical address to the management computer, as recited in 
independent claims 1, 2, 5, and 8. Nor does it disclose storing information described on a 
disk management table in a disk control unit from respective disk control imits together with 
identifiers of the disk control units into a storage device accessible by the management 
computer; requesting the disk control unit to which a volume corresponding to a requested 
group or a requested volume identifier belongs, to acquire volume list information concerning 
a volume belonging to the group or the volume identifier thus requested; and according to the 
volume list information as a response to the request, updating an information file in the disk 
management table stored in the storage device accessible by the management computer to the 
disk management table information thus notified, as recited in independent claims 9, 10, and 
11. 

2. U.S. Patent No. 6,529,944 Bl 

This reference discloses a host system for remote control of mass storage 
volumes using cascading commands with a selected identifier being in the command as 
issued from the host system. The command is recognizable by each mass storage system in 
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the stream of mass storage systems. The selected identifier is selected from a group of 
identifiers. The identifiers in the group identify the mass storage systems in the stream of 
mass storage systems. 

The reference does not teach registering, by the host computer, a volume 
identifier, a physical address, and a group identifier of the storage sub-system in a disk 
management table in the disk control units as a result of grouping of volumes in the storage 
sub-system according to a use purpose; obtaining, by the disk control unit, a physical address 
of a volume belonging to a same group as a group identifier input from the management 
computer from the disk management table, and notifying information described in a volume 
list of a volume allocated at the physical address to the management computer, as recited in 
independent claims 1, 2, 5, and 8. Nor does it disclose storing information described on a 
disk management table in a disk control unit from respective disk control units together with 
identifiers of the disk control units into a storage device accessible by the management 
computer; requesting the disk control unit to which a volume corresponding to a requested 
group or a requested volume identifier belongs, to acquire volume list information concerning 
a volimie belonging to the group or the volume identifier thus requested; and according to the 
volume list information as a response to the request, updating an information file in the disk 
management table stored in the storage device accessible by the management computer to the 
disk management table information thus notified, as recited in independent claims 9, 10, and 
11. 

3. U.S. Patent No. 4,985,828 

This reference discloses a method and apparatus for generating a real address, 
multiple virtual address spaces of a storage provides group identifiers in each of the entries in 
the address translation table, the address translation buffer and the entry in the address control 
register for identifying a respective area shared by a specific group of virtual spaces. See 
column 3, lines 12-67. 

The reference does not teach registering, by the host computer, a volume 
identifier, a physical address, and a group identifier of the storage sub-system in a disk 
management table in the disk control units as a result of grouping of volumes in the storage 
sub-system according to a use purpose; obtaining, by the disk control imit, a physical address 
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of a volume belonging to a same group as a group identifier input fi-om the management 
computer fi-om the disk management table, and notifying information described in a volume 
list of a volume allocated at the physical address to the management computer, as recited in 
independent claims 1, 2, 5, and 8. Nor does it disclose storing information described on a 
disk management table in a disk control xmit fi-om respective disk control units together with 
identifiers of the disk control units into a storage device accessible by the management 
computer; requesting the disk control unit to which a volume corresponding to a requested 
group or a requested volume identifier belongs, to acquire volume list information concerning 
a volume belonging to the group or the volume identifier thus requested; and according to the 
volume list information as a response to the request, updating an information file in the disk 
management table stored in the storage device accessible by the management computer to the 
disk management table information thus notified, as recited in independent claims 9, 10, and 
11. 

4. Japanese Patent PubUcation No, JP 2002-132552 

This reference relates to an information processing system capable of 
specifying not only data set name but also physical position, volume catalog only and volxmie 
name or data set as the units to set and release resident on a cache memory. In the system, a 
utility program 1 of resident control enables stay resident with issuing information of the 
physical position necessary to release resident to a disk array subsystem 13. For the 
specification of the volimae catalog/volume name, the program 1 acquires physical 
information necessary to release resident fi-om volume catalog information in a volume 14 in 
a disk-driving device 9. The program 1 acquires the physical position necessary to release 
resident by comparing resident control information 7 of dynamic cache resident mechanism 
with information of the volume catalog to delete spaces remaining on the cache memory. 

According to this conventional technique, a cache memory provided in the 
disk control unit of the storage device stores a volume list and a catalogue file as file 
allocation information. Moreover, physical region data is resident in the cache memory and 
the resident state can be released. The volume list is stored in the cache memory, which is 
not accompanied by the I/O operation to/fi-om the disk device of the storage device and 
accordingly, it becomes possible to acquire the volume list information with low overheads. 
Moreover, since the catalogue managing the entire system file information can be handled as 
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a type of file, the technique can acquire the catalog information with low overheads like the 
volume list. Present application at page 1, line 21 to page 2, line 12. 

This conventional technique has no consideration on increase of the overheads 
required for information acquisition due to increase of the number of devices accompanying 
the increase of the computer system size. Since the volume list exists on a storage device 
basis, a problem arises that as the number of storage devices used increases, the overheads 
required for information acquisition increase. Especially in this conventional technique, 
information is acquired via a host computer and there may arise a problem of deteriorating 
the throughput or response time of work executed in the host computer. The catalog file also 
has a problem that increase of the overhead required for information acquisition 
accompanying the catalogue file size is expected as the system size increases. Moreover, as 
the system size increases, not only the catalog for managing the entire computer system is 
generated but also a great nmnber of catalog files are generated according to the work and use 
purpose. The technique has a problem that such an increase of the nimiber of catalog files 
also causes the increase of overheads required for information acquisition. Present 
application, at page 2, line 15 to page 3, line 12. 

The reference does not teach registering, by the host computer, a volume 
identifier, a physical address, and a group identifier of the storage sub-system in a disk 
management table in the disk control units as a result of grouping of volumes in the storage 
sub-system according to a use purpose; obtaining, by the disk control unit, a physical address 
of a volume belonging to a same group as a group identifier input firom the management 
computer firom the disk management table, and notifying information described in a volume 
list of a volume allocated at the physical address to the management computer, as recited in 
independent claims 1, 2, 5, and 8. Nor does it disclose storing information described on a 
disk management table in a disk control unit fi-om respective disk control units together with 
identifiers of the disk control imits into a storage device accessible by the management 
computer; requesting the disk control unit to which a volume corresponding to a requested 
group or a requested volume identifier belongs, to acquire volume list information concerning 
a volume belonging to the group or the volume identifier thus requested; and according to the 
volume list information as a response to the request, updating an information file in the disk 
management table stored in the storage device accessible by the management computer to the 
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disk management table infomiation thus notified, as recited in independent claims 9, 10, and 
11. 

(f) In view of this petition, the Examiner is respectfully requested to issue 
a first Office Action at an early date. 



Respectfully submitted. 
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[57] ABSTRACT 

A storage control device is connected between a num- 
ber of access request control units and a storage device 
including a number of memory units. The apparatus 
includes a number of transmission units, each corre- 
sponding to one of the access request control unit. Each 
transmission unit receives access requests from its asso- 
ciated access request control unit and divides the access 
requests into a number of groups in the order of issuance 
from the access request control units. The transmission 
units also add access request identifiers to the access 
requests in each group and transmit a number of access 
requests with access request identifiers to a number of 
access request deciders. Each access request decider is 
associated with one of the independently accessible 
memory units. Each decider receives the access re* 
quests directed to its associated memory unit and sen- 
ally supplies the requests to that memory unit. Addi- 
tionally» a detection unit is connected to all of the access 
request deciders in order to detect that all access re- 
quests having the same access request identifier, as as- 
signed by the identifier transmission units, have been 
transmitted from the access request deciders to the 
corresponding memory units. 

10 Claims, 5 Drawing Sheets 
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STORAGE CONTROL METHOD AND 
APPARATUS 

BACKGROUND OF THE INVENTION 5 

This invention relates to storage control method and 
apparatus for a computer system, especially, being suit- 
able for operating a plurality of access request control 
units synchronously in parallel and ensuring the se- 
quence or order between access requests which are 
sequentially issued to a storage area. 

In conventional storage controlling, access requests 
are issued from a plurality of access request control 
units to a storage comprised of a plurality of indepen- 
dently accessible memory units (memory banks) as will 
be described below with reference to FIG. 1. 

Referring to FIG. 1, access request control units 20A 
to 20D are sources for issuing access requests. Access 
request stack units 21A to 21D respectively stack access 
requests issued from each of the access request control 
units 20A to 20D. Each of the stack units sends a 
stacked access request in the order of stacking, to one of 
access request priority deciders 22A to 22D in accor- 
dance with address information contained in that access 
request. A storage area 23 is comprised of memory 25 
banks 23A to 23D. 

Taking the access request control unit 20A, for in- 
stance, access requests issued from the unit 20A are 
stacked in a stack circuit 211 of the access request stack 
unit 21A and under thi. direction of a control circuit 30 
210, a stacked access request is sent to one of the access 
request priority deciders 22A to 22D corresponding to 
one memory bank which is designated by an address 
contained in that access request. Each access request 
priority decider 22A, 22B, 22C or 22D selects one of the 35 
access requests sent from the access request stack units 
21A to 21D and sends a selected access request to the 
storage area 23. The selection may be accomplished 
pursuant to priority grading which is predetermined 
among the access requests. The priority grading may be 40 
changed desirably to treat the access requests from the 
access request stack units as equally as possible. For 
example, the access request stack units 21A to 21D may 
be graded in the order of 21A. 21B, 21C and 21D for the 
initial concurrent access cycle, in the order of 21B, 21C, 45 
21D and 21A for the next concurrent access cycle and 
in the order of 21C, 21D, 21A and 21B for the concur- 
rent access cycle after next. 

In another example, the order of 21A, 21B, 21C and 
21D and the order of 21D, 21C, 21B and 21A may be 50 
repeated alternately for the purpose of the equal treat- 
ment. 

It is thus regulated that a plurality of access requests 
be applied to one decider, for example. 22A at a time 
and access requests be sent serially one by one from the 55 
decider to one memory bank, for example, 23A. 

Taking the access request priority decider 22A, for 
instance, a priority decision logic 220 checks and de- 
cides the access requests sent from the access request 
stack units 21A to 21D to the decider 22A for their 60 
priority and transmits a selected one of the access re- 
quests to the memory bank 23A of the storage 23. The 
other access requests not selected at that time are urged 
to wait at the entrance to the priority decision logic 220. 

The access requests are transmitted from the access 65 
request control unit 20A until the stack 211 of the access 
request stack unit 21A fills up. When a control circuit 
221 of, for example, the decider 22A transmits a signal 
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213 indicative of the fact that an access request 212 
issued during the preceding machine cycle (a periodical 
predetermined interval of time during which a group of 
sequential circuits constituting the system operate syn- 
chronously) is selected by the logic 220 and the stack 
unit 21A receives the signal 213, the succeeding access 
request 212 is sent from the stack unit 21 A. This war- 
rants that in the order of the access requests issued from 
the access request control unit 20A, accessed data ele- 
ments can be read out of the storage 23. 

In addition to the prior art storage control apparatus 
described above» another storage control apparatus has 
been proposed as disclosed in JP-A-No. 60-136849. 
according to which with a view to improve the perfor- 
mance of the entire system, access requests to be issued 
from an access request control unit are divided, in the 
order of issuance, into groups each having a access 
requests (a is an integer) in unit whereby a access re- 
quests in each group are respectively added with access 
request identifiers 0 to (a— 1) and then are issued from 
the access request control unit, and an access request 
priority decider directly coupled to a memory unit 
selects an access request and returns an access request 
identifier for the selected access request to the access 
request control unit which is an originator. 

Incidentally, a vector processor for fast processing of 
a scientific computation comprises a plurality of vector 
registers for holding vector data, a plurality of arithme- 
tic units for operating on the data, and a plurality of 
access request control units for data transfer between a 
storage and each of the vector registers, whereby vec- 
tor elements in one vector instruction are concurrently 
allocated to a plurality of resources, such as vector 
registers, arithmetic units or access request control 
units, and are processed in parallel. Such parallel pro- 
cessing is a so-called element parallel processing mode 
and has been employed frequently in vector processors. 

Generally speaking, it is desirable that the resources 
in the same group operating concurrently in the element 
parallel processing mode by completely synchronized 
with each other to process the allocated elements. By 
the complete synchronization, a control circuit can be 
used in conmion for the plurality of resources in the 
same group operating in parallel, thereby simplifying 
the control logic. In this approach, there arises however 
a problem that a waiting time takes place owing to 
competition for accessing to memory bank constituting 
the storage. Accordingly, in order for the plurality of 
resources in the same group to operate synchronously 
with each other, the advent of a storage control appara- 
tus is desired which can absorb asynchonism occurring 
between the resources on account of the waiting time to 
thereby completely synchronize the storage accessing. 

Reviewing then the prior art storage control appara- 
tus disclosed in JP-A-No. 60-136849 mentioned previ- 
ously, it will be seen that this prior art apparatus presup- 
posed the fact that one access instruction is processed by 
allocating it to a single access request control unit and 
fails to take into consideration a processing in which 
data elements treated by one vector access instruction 
are divided for allotment to a plurality of access request 
control units with the aim of being processed in parallel, 
thus resulting in the problem that the plurality of access 
request control units to be operated in parallel can not 
be synchronized with each other so as to process access 
requests. 
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SUMMARY OF THE INVENTION 
An object of this invention is to provide a storage 
controller capable of synchronizing a plurality of access 
request control units with each other to enable the plu- 5 
ral access request control units to issue access requests 
every clock pitch, in a vector processor or the like in 
which vector data elements treated by one vector ac- 
cess instruction are divided for allocation to the plural- 
ity of access request units and they are then processed. 10 

Another object of this invention is to provide an 
element parallel process capable of completely sup- 
pressing, within a storage controller, asynchronism in 
access request processing time due to storage accessing 
which mainly causes disturbance of the synchronous 15 
operation for data elements. 

Another object of this invention is to provide storage 
control apparatus and method capable of receiving 
access requests, issued concurrently from a plurality of 
access request control units, every clock pitch to 20 
greatly improve ability to process the access requests by 
synchronizing them with each other. 

Still another object of this invention is to provide a 
storage control apparatus and method, based on an 
element parallel processing mode in which vector ele- 25 
ments treated by one vector access instruction and di- 
vided for allocation to a plurality of access request 
control units with the aim of being processed at a time, 
which can divide access requests issued from the plural- 
ity of access request i^imtrol units into groups each 30 
having a access requests in unit and can exchange the 
order of issuance of the a access requests in unit to a 
storage while warranting the sequence of read vector 
data elements. 

To accomplish the above objects, according to the 35 
invention, access requests issued from a plurality of 
access request control units are added with access re- 
quest identifiers, signals are produced when it is de- 
tected that access requests with the same access request 
identifier issued from a plurality of access control units 40 
adapted to process one access instruction are all se- 
lected by a plurality of access request priority deciders, 
and the detection signals are used to permit issuance of 
the succeeding access requests with the same access 
request identifier, while a plurality of read data ele- 45 
ments corresponding to the identifiers are transmitted 
concurrently to the access request originators after 
expiration of a fixed time required for the detection 
signals to access a storage. 

Especially where the access operation to a series of 50 
elements of vector-like data is processed by dividing the 
vector data elements for allocation to a plurality of 
access request control units, access requests to be issued 
from respective access request control units are divided, 
in the order of request issuance, into groups each having 55 
a access requests in unit and the a access requests in 
each group are respectively added with access request 
identifiers 0 to (a — 1), while a plurality of access request 
priority deciders respectively coupled to memory units 
select all access requests with the same access request 60 
identifer issued from the plurality of access request 
control units adapted to division process one access 
instruction and return the access identifiers for the se- 
lected access requests to the plurality of access request 
control units which are originators. This ensures that 65 
each of the plurality of access request control units for 
division processing one instruction can perform sequen- 
tial issuance of at least a access requests regardless of 



whether the a access requests are selected by the access 
request priority deciders. Further, when access requests 
(with identifiers "0") issued, during the first issue cycle, 
from the plurality of access request control units 
adapted to division process one instruction are all se- 
lected by the deciders and the signals to this effect are 
then returned to the access request control units, the 
access request control units are allowed to issue access 
requests with identifiers "0" for the second issue cycle 
following issuance of access requests with the previous 
identifiers "a — T' for the first issue cycle. The succeed- 
ing issue cycles are taken care of in a similar way so that 
the access request control units for division processing 
one instruction can be synchronized with each other to 
sequentially issue access requests. 

In a read data controller for transmitting read data 
elements corresponding to access requests to the corre- 
sponding access request originators, identifiers atten- 
dant on read data elements are recognized and data 
elements are stored in buffer positions corresponding to 
the identifiers. When a fixed time (access time of the 
storage) expires following detecting that all of the ac- 
cess requests with the same identifier issued from the 
plurality of access request control units adapted to divi- 
sion process one instruction are transmitted to the stor- 
age, that is, at the time that data elements of the same 
identifier are all stored in the read dau buffers, the data 
elements stored in the buffers are transmitted synchro- 
nously in parallel, in the order to identifiers, to the data 
originators. This ensures that a plurality of read data 
elements corresponding to access requests issued from 
the plurality of access request control units adapted to 
division process one instruction can be transmitted 
while warranting the sequence of the read data ele- 
ments. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram showing a prior art storage 
control apparatus. 

FIG. 2 is a block diagram showing the essential part 
of a computer system employing a storage controller 
according to the invention. 

FIG. 3 is a circuit diagram showing a storage control- 
ler according to an embodiment of the invention. 

FIG. 4 is a circuit diagram showing an embodiment 
of an access request identifier control circuit shown in 
FIG- 3. 

FIG. 5 is a circuit diagram showing an embodiment 
of a read data transmission control circuit shown in 
FIG. 3. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

The invention will now be described by way of exam- 
ple with reference to the accompanying drawings. 

FIG. 2 illustrates, in block form, an exemplary con- 
struction of the essential part of a computer system 
incorporating an embodiment of the invention. The 
computer system comprises an arithmetic section 30 
including a plurality of (four in this embodiment) arith- 
metic units 30A to 30D, a vector register section 31 
including vector registers 31 A to 31D and adapted to 
serve as a data buffer between the arithmetic section 30 
and a storage 15, an access request controller 32 includ- 
ing access request control units 32A to 32D, and a Stor- 
age conUoller 10. The storage 15 includes a plurality of 
(four in this embodiment) independently accessible 
memory banks 15A to 15D. Address information atten- 
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dant on an access request is decoded to determine 
which memory bank should be accessed. The storage 
controller 10 includes access request stack circuits lOA 
to lOD in association with the access request control 
units 32A to 32D, read data buffer circuits 14A to S 
and access request priority deciders llA to IID in 
association with the memory banks 15A to 15D. 

The operation of the computer system shown in FIG. 
2 will be described below by referring to, as an example, 
a storage read operation based on the element parallel 10 
processing mode. 

Firstly, when reading a piece of vector data from the 
storage and storing the vector data in the vector register 
unit 31, vector elements are allocated to the access 
request control units 32A to 32D in the following man* IS 
ner and access requests are generated. 

Access request 

Control Units 

32A: 0th, 4th, 8th, . . . ^th elements 20 
32B: 1st, Sth, 9th, . . . (4ii+l)th elements 
32C: 2nd, 6th, 10th, . . . (4n+2)th elements 
32D: 3rd, 7th, 11th, .. . (4/t+3)th elements (n being 
positive integer) 

Access requests concurrently generated for four ele* 25 
ments (for example, the 0th, 1st, 2nd and 3rd elements) 
are respectively sent to the corresponding access re- 
quest stack circuits lOA to lOD. In accordance with an 
address contained in an access request, each stack cir- 
cuit sends that access request to one of the priority 30 
deciders llA to IID. If a plurality of access requests 
compete with one another in either of the priority de- 
ciders, that priority decider selects one access request 
pursuant to predetermined priority grading and then 
sends the selected access request to corresponding one 35 
of the memory banks 15A to 15D. The read data corre- 
sponding to the access requests sent to the memory 
banks are read and transferred to the storage controller 
10 after expiration of a Hxed time (corresponding to an 
access time), so that elements of the read data represen- 40 
tative of the vector elements are set to the read data 
buffer circuits 14A to 14D corresponding to the access 
request control circuits 32A to 32D. When all the data 
elements corresponding to the four access requests con- 
currently issued from the acx^ess request control circuits 4S 
32A to 32D are read out of the storage IS, these read 
data elements are sent to the access request control 
units. This data group is sent in the order of the concur- 
rent issuance and then stored concurrently in the vector 
roisters 31A and 31D. Allocation of the vector ele- 50 
ments to the vector register unit 31 is as follows. 
Vector registers 

31A: 0th, 4th, 8th, . . . 4nth elements 
31B: 1st, Sth, 9th, . . . (4/i-hl)th elements 
31C: 2nd, 6th, 10th, . . . (4rt-h2)th elements 55 
31D: 3rd, 7th, 1 1th, . . . (4/i + 3)th elements 
For arithmetic operation of the data stored in the 
vector registers 31 A to 31 D, the vector elements are 
allocated to the arithmetic units 30A to 30D in the 
following manner and operation results are again stored 60 
in the vector registers. 
Arithmetic units 

30A: 0th, 4th, Sth. . . . 4nth elements 
30B: 1st, Sth, 9th, . . . (4n-f /)th elements 
30C: 2nd, 6lh. 10th, . . , (4rt + 2)th elements 65 
30D: 3rd, 7th, lUh, . . . (4rtH-3)th elements 
In the above arithmetic operation, the four arithmetic 
units 30A to 30D arc completely synchronized with 
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each other for operation. For example, results for the 
0th, 1st, 2nd and 3rd elements are obtained concurrently 
and stored in the vector registers 31 A to 3 ID. 

When writing the data stored in the vector registers 
31A to 31D into the storage 15, the vector elements are 
allocated to the access request control units 32A to 32D 
as in the case of data read previously described and four 
elements, for example, the 0th, 1st, 2nd and 3rd elements 
are sent to the corresponding access request stack cir- 
cuits lOA to lOD. Thereafter, access requests are sent to 
the storage 15 similarly to the read operation. 

As described at>ove, the four arithmetic units 30A to 
30D are synchronized with each other, the four vector 
registers 31A to 31D are synchronized with each other 
and the four access request control units 32A to 32D are 
synchronized with each other, so as to process the vec- 
tor elements. Accordingly, the element parallel process- 
ing mode based on synchronous operation is permitted 
to take a logical construction which uses a single con- 
trol system logic to control a set of the four arithmetic 
units 30A to 30D, a set of the four vector registers 31A 
to 31D and a set of the four access request control units 
32A to 32D. 

Within the storage controller 10, however, the four 
access requests concurrently issued from the access 
request control units 32A to 32D operating in synchro- 
nism are not always processed concurrently because of 
a condition of an accessed memory bank (for example, 
the memory bank being occupied by the preceding 
access request) and competition with another access 
and as a result, the access results are sent to the memory 
banks at different times. Accordingly, a control system 
is needed wherein the read data buffer circuits 14A to 
14D of the storage controller 10 wait until the read data 
elements corresponding to the access requests concur- 
rently sent from the access request control units 32A to 
32D are all stored in the buffers 14A to 14D and when 
all the data elements are stored, the data elements are 
sent to the four read data access request control units 
32A to 32D at a time. 

A synchronous control mode in the storage control- 
ler will now be described in greater detail with refer- 
rence to FIG. 3. 

The construction of the storage controller 10 of FIG. 
2 is exemplarily detailed in FIG. 3, along with connec- 
tion to the memory banks 15A to 15D. In addition to the 
access request stack circuits lOA to lOD, priority decid- 
ers 11 A to IID and read data buffer circuits 14A to 
14D, the storage controller 10 comprises read data 
transfer control circuits 12A to 12D and an access re- 
quest identifier control circuit 13. 

Four access requests for, for example, the 0th, 1st, 
2nd and 3rd dements issued concurrently from the 
access request control units 32A to 32D arrive at the 
access request stack circuits lOA to lOD, respectively. 
The access request reaching the access request stack 
circuit lOA, for example, is set in one of stacks SO 
(lOOA), S2 (lOOB), S2 (lOOC) and S3 (1W)D), for exam- 
pie, the stack SO (lOOA) which is designated by an input 
control circuit 104. The input control circuit 104 desig- 
nates one of the stacks by sending thereto a signal 104a 
indicative of one of stack positions **0'* to "3** (corre- 
sponding to the stacks SO to S3) to which one access 
request is to be stored. The indication of the signal 104a 
changes to circulate through 

"0**— *"l"— *'"2"— ♦"3"— *"0" . . . each time that one ac- 
cess request is stored in one stack, thereby designating a 
subsequent store stack position. 
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Each access request stored in one of the stacks SO to 
S3 of each of the access request stack circuits lOA to 
lOD is sent to the priority deciders 11 A to IID through 
a selection circuit 103 under the control of an output 
control circuit 105, For example, when the output con- 
trol circuit 105 designates the stack position or number 
representative of the stack SO (lOOA), an access 
request stored in the stack SO (lOOA) is delivered. Like 
the input control circuit 104, the output control circuit 
105 designates one of the stacks by sending to the selec- 
tion circuit 103 a signal 105a indicative of one of stack 
positions '*0'* to "3" from which one access request is to 
be delivered. The indication of the signal 105i2 changes 
to circulate through ••0*W»1"— "2"-*"3"->^'0" . . . 
each time that one access request is delivered to the 
selection circuit 103. However, the output control cir- 
cuit 105 is different from the input conuol circuit 104 in 
that it controls the delivery of the access request from 
the designated stack dependent on contents of access 
request transmission control flip-flops lOlA to lOlD 
associated with the stacks SO to S3, respectively. 

More particularly, when the output control circuit 
105 accesses, for example, the access request transmis- 
sion control flip-flop lOlA associated with the stack SO 
(lOOA) and the content of the flip-flop 101 A is "1**, an 
access request stored in the stack SO (lOOA) is delivered 
to the {Priority deciders llA to IID. Contrarily, when 
the content of the flip-flop llA is •'0". the delivery of 
the access request from the stack SO (lOOA) is inhibited 
until the flip-flop 101 A - set to by a signal 13a from 
the access request identifier control circuit 13 and at the 
same time, the stack number "0** indicated by the signal 
105a deUvered out of the output control circuit 105 is 
maintained. 

Where the content of the flip-flop lOlA is "1", after 35 
the access request stored in the assodiated stack SO 
(lOOA) has been fetched and delivered to the priority 
deciders llA to IID, the output control circuit 105 
resets the flip-flop lOlA to "0" by sending thereto a 
signal WSb representative of information about comple- 
tion of that delivery. Then, the output control circuit 
105 is ready to access the subsequent stack SI. 

Taking the access request stack circuit lOA, for in- 
stance, an access request 103a sent from the selection 
circuit 103 to the priority deciders 11 A to IID contains, 
in addition to address information stored in for example 
the stack SO (lOOA), one of access request identiflers of 
two digits "00", **02** and **03" delivered out of 

access request identifier generators 102A to 102D, the 
first digit *'0" of the access request identifier being in- 
dicative of the access request stack circuit lOA in ques- 
tion and the second digit **0", "I", "2" or "3" being the 
stack number designating the stack SO, SI, S2 or S3. In 
this example, the access request identifier "00** is desig- 
nated. 

Although the access request stack circuit lOA has 
been described as an example, each of the remaining 
stack circuits lOB to lOD operates similarly to the stack 
circuit lOA. Thus, four access requests issued concur- 
rently from the four access request control units are 
stored and processed in the corresponding stacks of the 
same number of respective stack circuits. 

The access request 103a sent from the access request 
stack circuit lOA can reach one of the priority deciders 
llA to IID in association with the memory banks, for 
example, the decider 11 A in accordance with an access 
address designated by that access request. The access 
request 103a reaching the priority decider 11 A is 
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checked for its priority in relation to another access 
request sent from the remaining access request stack 
circuit lOB, IOC or lOD and if selected, an access re- 
quest llAa is sent to the corresponding memory bank 
ISA, together with information indicative of either read 
or write mode, an address on the memory and data in 
the case of write mode. 

Concurrently with the delivery of the access request 
11 Aa, an access request identifier llA^ is sent to the 
access request identifier control circuit 13. Details of 
the access request identifier control circuit 13 are illus- 
trated in FIG. 4. Referring to FIG. 4. the access request 
identifier 11A6 is supplied to a decoder circuit 40A 
which includes a decoder 41 for decoding a number 
representative of the access request stack circuit lOA, 
lOB, IOC or lOD and decoders 42A to 42D for decoding 
a stack number in each circuit lOA, lOB, IOC or lOD 
and which is operative to specify the access request 
stack circuit and the stack number. Signals from the 
decoder circuit 40A are passed through OR gates 430 to 
43F so as to set to "1" one of flip-flops 440 to 44F which 
are adapted to indicate validity of access request identi- 
fier for each access request stack circuit specified by the 
decoder circuit. Similarly, access request identifiers 
IIB^ to 11D6 are decoded by decoder circuits 40B to 
40D and used to set specified flip-flop for indication of 
validity of access request identifier. 

When the flip-flop corresponding to the stacks of the 
same number in respective access request stack circuits 
lOA to lOD, for example, the flip-flops 440, 444, 448 and 
44C are all rendered **r% indicating that the four access 
requests reaching the access request stack circuits lOA 
to lOD at a time and set in the stacks SO in respective 
stack circuits are all sent to the storage 15, all of the "1" 
signals are detected by an AND gate 45A (similar AND 
gates 45B to 4SD are provided) to produce the signal 
13a which sets to "T* the contents of the access request 
transmission control flip-flops lOlA associated with the 
stacks SO in respective access request stack circuits lOA 
to lOD. This ensured that the subsequent access re- 
quests stored in the stacks SO in respective access re- 
quest stack circuits lOA to lOD are ready for transmis- 
sion to the priority deciders 11 A to IID. At the time the 
signal 13a is sent to the access request stack circuits lOA 
to lOD, the validity indication flip-flops 440, 444, 448 
and 44C producing the signal 13a are reset to "0**. 

Incidentally, read data elements 15Aa to 15Da corre- 
sponding to the access requests llAa to llDa sent to 
the memory banks ISA to 15D are transferred to the 
read data bufier circuits 14A to 14D, together with 
access request identifiers 12Aa to 12Da retarded at the 
read data transfer control circuits 12A to 12D by an 
access time of the memory banks and the stack numbers. 
On the basis of a stack number and an access request 
stack circuit number designated by the access request 
identifier, each of the read data buffer circuits 14A to 
14D stores, under the direction of a selection circuit 
140, one of the read data elements into one of data buff- 
ers BO (141A) to B3 (141D) of each of the read data 
buffer circuits 14A to 14D respectively corresponding 
to the access request stack circuits 10 A to lOD, the data 
buffers being provided in corresp>ondence to the stacks 
SO to S3 of each of the access request stack circuits 10 A 
to lOD. For example, the identifier 12Aa indicative of 
"00'* serves to store the read data element ISAa in the 
data buffer BO (141A) of the read data buffer circuit 
14A. In the event that the number of access requests 
designated by an access instruction is too small to oc- 
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cupy all of the memory banks, some of the access re- 
quests llAo to llDa sent to unoccupied memory banks 
are added with non-operation codes to thereby com- 
plete the access instruction. 

Incidentally, a read data transmission control circuit 3 
16 operates independently of the storage of the read 
data elements to the data buffer circuits to concurrently 
transmit the four read data elements sorted in the read 
data buffer circuits 14A to 14D to the access request 
control circuits 32A to 32D. FIG. 5 illustrates details of 10 
the read data transmission control circuit 16. 

Sent to the read data transmission control circuit 16 
are the signals 13a to 13d from the access request identi- 
fier control circuit 13 respectively indicating that access 
requests stored in the stacks of the same number in 
respective four access request stack circuits lOA to lOD 
are all transferred to the memory banks. Each of the 
signals 13a to 13^^ indicates, in other words, that read 
data elements are all stored in corresponding one of the 
four data buffers BO (141A) to B3 (141D) in respective 
four read data buffer circuits 14A to 14D after expira- 
tion of a fixed time (corresponding to the access time of 
each memory bank). Accordingly, validity of the read 
data elements in corresponding one of the data buffers 
BO (141A) to B3 (141D) in respective read data buffer 
circuits 14A to 14D can be indicated by passing the 
signals 13a to 13d through buffers 54A to 540 which 
delay these signals to match with the memory bank 
accessing so as to set fl»i>-nop 50 A to SOD. 

The read data transmission control circuit 16 is opera- 
tive to sent to the read data buffer circuits 14A to 14D 
a control signal 16a which controls sequential delivery 
of the valid data elements from the read data buffers, 
beginning with the data buffer BO (141 A) followed by 35 
the data buffers Bl (141B), B2 (141C) and B3 (141D). 
But when the content of a flip-flop for indication of read 
data validity 50A, SOB, 50C or SOD corresponding to a 
data buffer designated by the content of read data 
pickup pointer 54 which is decoded by a decoder 55 is 4Q 
**0'*, that is, when a read data element corresponding to 
at least one of four access requests which is stored in a 
stack within the access request stack circuits lOA to 
lOD, the position of that stack being designated by the 
pointer 54, has not reached the read data buffer yet, the 45 
delivery of the read control signal 16a is inhibited by 
AND gates 51 A to 51 D and the content of the pointer 
54 also remains unchanged until that flip-flops 50 A, 
SOB, 50C and SOD are set to Subsequently, when 
the content of the flip-flop for indication of read data 50 
validity corresponding to the data buffer designated by 
the pointer 54 changes to **V\ this change is detected by 
the AND gates 51A to SID and an OR gate 52, thereby, 
enabling the read control signal 16a to be transmitted 
through a flipflop 53 A. Concurrently therewith, a sig- 55 
nal 52a resets the read data validity indication flip-flop 
designated by the pointer 54 to *'0" and the content of 
the pointer 54 is renewed by being added with "-hi" 
from a -f 1 circuit 56. 

The four read data buffer circuits 14A to 14D receive 60 
the read control signal 16a simultaneously so that the 
read control signal 16a is applied to an output control 
circuit 143 provided in each circuit 14A, 14B, 14C or 
14D. The output control circuit 143 then issues a signal 
143a which controls a selection circuit 142 such that a 65 
read data element stored in a data buffer whose number 
or position is designated by the read control signal 16a 
can be picked up. 
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Through the operation described above, four read 
data elements can be transmitted synchronously in par- 
allel from the four read data buffer circuits 14A to 14D 
to the access request control units 32A to 32D. 

As has been described, according to the present em- 
bodiment, of four access requests issued concurrently 
and synchronously from the four access request control 
units 32A to 32D^ a maximum of three access requests 
can be overridden within the storage controller 10 so 
that the four access requests can be processed every 
clock pitch, and besides read data elements correspond- 
ing to the four access requests received at a time can be 
sent to the access request control units 32A to 32D in 
the order of the synchronous, parallel receipt of the 
access requests. 

I claim: 

1. A storage control apparatus, connected between a 
storage device, said storage device including a plurality 
of independently accessible memory units, and a plural- 
ity of access request control units, each of said plurality 
of access request control units being operative to issue, 
to said storage device, access requests corresponding to 
elements of data, said storage control apparatus sequen- 
tially transmitting selected ones of said access requests 
issued from said plurality of access request control units 
to corresponding memory units designated by said ac- 
cess requests, said storage control apparatus compris- 
ing: 

a plurality of identifier transmission means for divid- 
ing said access requests issued from said plurality of 
access request control units into a plurality of 
groups based on the order of issuance from said 
access request control units wherein each identifier 
transmission means is associated with one of said 
access request units and divides said access requests 
from said associated access request unit, and 
wherein each identifler transmission means further 
adds access requests in each group with access 
request identiflers, and wherein each identifier 
transmission means further transmits a plurality of 
access requests with access request identifiers 
added thereto; 

a plurality of access request decider means, each 
access request decider means being provided in 
correspondence to one of said plurality of memory 
units, for receiving the plurality of access requests 
with access request identifiers sent from said identi- 
fler transmission means and serially supplying to 
the corresponding memory units the received ac- 
cess requests one by one; and 

means, connected to said plurality of access request 
decider means, for detecting that all access requests 
with the same access request identifier have been 
transmitted from said plurality of access request 
decider means to said corresponding memory units, 
said means for detecting further sending to all of 
said identifler transmission means information in- 
dicative of the detection that all access requests 
with the same access request identifler have been 
transmitted from said plurality of access request 
decider means to said corresponding memory units 
which permits said same access request identifier to 
be used for new access requests. 

2. The storage control apparatus of claim 1 further 
comprising: 

a plurality of buffer means, each one of said buffer 
means being associated with one of said plurality of 
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memory units, for holding respective data elements 
read out of respective memory units; and 
means, receiving from said plurality of identifier 
transmission means signals indicative of the fact 
that all of the access requests having the same ac- 5 
cess request identifier have been detected, for in- 
forming said plurality of buffer means of said detec- 
tion indicative signals so as to permit said read data 
elements held in said plurality of buffer means to be 
transmitted to said plurality of access request con- !0 
trol units. 

3. The storage control apparatus of claim 2 wherein 
said each identifier transmission means comprises a 
stack for access requests for said data elements received 
from said associated access request control unit, and a 15 
plurality of flip-flops associated with each stack for 
controlling whether the access requests held in said 
stacks are permitted to be transmitted to said access 
request decider means, and wherein said plurality of 
buffer means each have a plurality of buffer positions 20 
for holding said data elements, and wherein each said 
stack is partitioned into a number of portions wherein 
said number of portions correspond to the number of 
access request identifiers, each portion corresponding 

to one of said identifiers, each one of said plurality of 25 
flip-flops being associated with one of said portions of 
said stacks and each buffer position being associated 
with one of said access request identifiers. 

4. The storage control apparatus of claim 1 wherein 
each said access reques* decider means is coupled to all 30 
of said plurality of identifier transmission means and 
wherein each said access request decider means com- 
prises priority means for serially delivering a plurality 

of access requests that are competing with each other to 
said memory units in accordance with a predetermined 35 
priority value. 

5. The storage control apparatus of claim 4 wherein 
said priority means comprises means for sequentially 
changing the priority value at the rate of a predeter- 
mined number of data access operations. 40 

6. The storage control apparatus of claim 2 further 
comprising means for sending, from respective access 
request decider means to respective buffer means, an 
access request identifier and an identification number, 
which identifier number identifies the identifier trans- 45 
mission means responsible for a given access request, in 
synchronization with a data element to be read out of 
the memory unit associated with said access request 
decider means wherein said data element is read out in 
response to said given access request. 50 

7. The storage control apparatus of claim 6 wherein 
said means for informing comprises means for providing 
a plurality of signals, one associated with each of said 
buffer means, with a delay time corresponding to an 
interval of time within which data elements have been 55 
read out of respective memory units and then trans- 
ferred to respective buffer means, said plurality of sig- 
nals indicating that all access requests having the same 
request identifier have been sent to said memory units. 

8. The storage control apparatus of claim 1 wherein 60 
said detecting means comprises, 

a plurality of latch means for latching a flag indica- 
tive of the validity of a detected identifier, the 
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number of said latch means being equal to the prod- 
uct of the number of said identifier transmission 
means and the number of access request identifiers 
in each identifier transmission means, 
decoders connected to said plurality of latch means, 
for discriminating the identification of said identi- 
fier transmission means based on said identification 
number, and 

decoders for discriminating individuality of said ac- 
cess request identifiers. 

9. A computer system comprising: 

a storage device having a plurality of independently 
accessible memory units; 

a plurality of access request control units operative to 
issue, to said storage device, access requests corre- 
sponding to elements of vector data; 

a plurality of identifier transmission means, each iden- 
tifier transmission means being associated with one 
of said plurality of access request control units, for 
dividing a plurality of access requests received 
from said access request control units into a plural- 
ity of groups based on the order of reception, 
wherein each said identifier transmission means 
transmits the plurality of access requests together 
with access request identifiers for identifying indi- 
viduality of said access requests; 

a plurality of access request decider means each asso- 
ciated with one of said memory units, for selec- 
tively receiving said access requests in accordance 
with addresses in said storage device and serially 
supplying to the corresponding one of said memory 
units the selected access requests one by one; and 

means, connected to the output of all of the plurality 
of access request decider means, for detecting that 
access requests with the same access request identi- 
fier have all been transmitted from said plurality of 
access request decider means to said address corre- 
sponding memory units in said storage. 

10. A storage control method for controlling an ac- 
cess sequence to respective elements of vector data in 
an orderly manner in a computer system including a 
storage device, having a plurality of independently 
accessible memory units, and a plurality of access re- 
quest issue units for issuing access requests to said stor- 
age device, said access requests corresponding to the 
vector data elements, said method comprising the steps 
of: 

dividing the access requests corresponding to the 

elements into a plurality of groups based on the 

order of request issuance; 
adding access requests in each group with access 

request identifiers which are common to respective 

groups; 

receiving access requests with the same identifier sent 
to respective memory units and deciding whether 
all of the access requests with the same identifier 
have been sent to respective memory units; and 

allowing, when the access requests with the same 
identifier have been sent to respective memory 
units the use of the same identifier for new vector 
data elements. 

« • * * * 
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(57) ABSTRACT 

A host system for remote control of mass storage volumes 
using cascading commands which collect information about 
linked remote volumes located in physically separate sites so 
other cascading commands can be issued to effect changes 
in linked remote volumes. A host computer program issues 
the cascading commands which ask the locally communi- 
cating mass storage system to return information which can 
be used to identify one or more levels of remote mass 
storage systems in a stream of remote mass storage systems. 
Once a mass storage system at a given level has been 
identified, commands can be sent by the host through the 
locally communicating mass storage system to cause actions 
to occur at the identified remote level, whether or not there 
are multiple intervening levels of remote mass storage 
systems. In one embodiment, a host computer can query, 
establish, split, re-establish, copy, restore, reverse split, 
differentially split and reverse differentially split business 
continuance volumes at any level in a stream of local and 
remote mass storage system sites. 
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HOST SYSTEM FOR REMOTE CONTROL 
OF MASS STORAGE VOLUMES USING 
CASCADING COMMANDS 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates generally to the field of mecha- 
nisms for allowing remote control to occur between host 
computers and a plurality of mass storage biisiness continu- 
ance volumes and more particularly to cascading commands 
for issuance by a host system to collect information about 
and transmit control commands to volumes at one or more 
levels away from the host in the system. 

2. Background 

Mass storage systems have become increasingly cost 
effective for critical business systems. Advances such as 
Redundant Arrays of Independent Disks (RAID) technolo- 
gies and Hierarchical Storage Management (HSM) systems 
have greatly improved the reliability of mass storage by 
providing a number of different redundancy features. 
Additionally, HSM systems such as the SYMMETRIX™ 
systems that are commercially available from the Assignee 
of the subject application provide disaster recovery facilities 
such as Symmetrix Remote Data Facilities (SFDF). These 
allow a SYMMETRIX'^'^ 5xxx system located at one site to 
maintain a continuous copy or mirror of the data at the 
logical volume level in other SYMMETRIX""^ systems 
located in physically separate sites. FIG. la (Prior Art) 
illustrates a redundancy technique used in SYMMETT^IX™ 
systems to provide mirroring, RAID configurations, and 
other forms of redundant disk storage. As seen in FIG. la 
(Prior Art) disk adapters DAI, DA2 and DA3 are connected 
over small computer storage interface (SCSI) buses to 
physical disk drives C, such as CI, C2 and C3 on disk 
adapter DAI. In SYMMETRIX'^" systems, in some 
implementations, a physical disk CI is divided into three 
logical didcs, called HI, H2, and H3. 

To illustrate this, assume a typical physical disk connected 
to a mainfirame computer contains 2000 cylinders. In an 
HSM system such as SYMMETRIX™ systems, shown in 
FIG. la (Prior Art) using disks C which are larger in capacity 
than the typical disks, the larger disks C can be logically 
divided into smaller logical units. If the disks C in this 
example hold 6000 cylinders, this physical disk C has the 
capacity of three typical disks. Each logical disk H, in this 
example, would be the equivalent of one typical disk. 

Still in FIG. la (Prior Art), if a typical disk contains a 
single large file or dataset named DSNl, mirroring redun- 
dancy techniques used in an HSM such as SYMMETRIX™ 
systems, can create copies of this dataset DSNl on disk 
adapters DA2 and DA3. In the example here, the first 
standard copy of DSNl is physically located on disk adapter 
DAI, disk CI, at logical disk H2. The first mirror copy, 
DSNIMI is located physically on disk adapter DA2, at 
physical disk C2, logical disk HI. The second mirror copy 
DSN1M2 is located on disk adapter DAB, physical disk C3, 
logical disk H3. 

In FIG. Ih (Prior Art), a more abstract way of thinking 
about mirroring or redundancy is shown. If the HSM system 
allows three mirrors for a standard disk, the HSM might 
have disks configured as shown here — the standard disk for 
data DSNl, is allocated to disk adapter DAI, physical disk 
CI. The first mirror, Mirrorl, is assigned to disk adapter 
DA2, physical disk C3, and so on. In SYMMETRIX'^'^ 
systems, the combination of disk adapter DA, physical disk 
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C and logical disk H, is resolved into a SYMMETRIX™ 
device number. In this example, the SYMMETRIX™ sys- 
tem synchronizes the mirrors in a transparent manner. When 
the data from disk adapter DAI, physical disk CI has been 
5 copied to mirrorl and mirror 2, the devices are considered 
synchronized. 

Now turning to FIG. 7c (Prior Art), disk adapters DA are 
shown as they might be configured in Symmetrix Remote 
Data Facilities (SRDF) systems for disaster recovery. As 

10 seen in FIG. 7C (Prior Art), a SYMMETRIX A(MASTER) 
system has been configured in a unidirectional SRDF cam- 
pus solution with SYMMETRIX B(SLAVE) system. 

When the SRDF features are used, a SYMMETRIX™ 
system includes not only cache memory, channel directors 
CD and disk adapters DA, but remote link directors RLD. 
Within each SYMMETRIX^^ ^nit, three volume types may 
be defined: local (L), source (Rl) and target (R2). Local 
volumes L are volumes unique to that SYMMETRIX™ unit. 
They are only accessible to hosts attached to that SYMME- 

20 TRIX™ unit (in this example, HOST 1.) 

Still in FIG. 7c (Prior Art), source volumes Rl are logical 
volumes that reside on a SYMMETRIX™ unit with the 
SRDF features activated, so that the data on source volumes 
Rl is mirrored or copied to respective target volumes R2 on 
another SYMMETRIX™ unit (in this example, HOST 2). 
The target volumes R2 are located on one or more separate 
SYMMETRIX™ units in an SRDF configuration. 

As seen in FIG. 7c (Prior Art), a path is established by the 
remote link directors RLD to allow data to be mirrored. The 
paths shown here are labeled remote access group 0, or RAO. 

Turning now to FIG. Id (Prior Art), a bidirectional SRDF 
configuration is shown. Host 1 is logically in communica- 
tion with standard volume std, in this example. As men- 

35 tioned above, SYMMETRIX™ systems would normally (in 
the absence of SRDF features) establish some mirroring for 
an ordinary volume. In this case, mirrors Ml and M2 in 
SYMMETRIX™ A might be established for standard vol- 
ume std. The SRDF feature takes this mirroring one step 

40 further. Instead of creating a mirror of SRDF standard source 
volume Rl on SYMMETRIX™ A, the SRDF features, using 
the remote link directors RLD, assign a remote mirror Ml in 
SYMMETRIX™ B. In other words, source volumes Rl are 
standard volumes that reside on a SYMMETRIX™ unit, 

45 with the data on those volumes mirrored to respective target 
R2 volumes on another SYMMETRIX™ unit, here SYM- 
METRIX™ B. If the source volume Rl fails, the SYMME- 
TRIX™ A will transparently access the data on the corre- 
sponding target volume R2 in SYMMETRIX™ B, When the 

5Q failing volume is replaced, the remotely mirrored pair is 
re-synchronized automatically as a backgroimd operation, 
using the data in the target volume R2. 

Still in FIG. Id (Prior Art), target R2 volumes are a copy 
of the source Rl volumes in another SYMMETRIX™ unit. 

55 A target volume R2, typically has a default configuration 
mode of "read-only" to any host with access to the SYM- 
METRIX™ unit in which it resides. In this example, mirror 
Ml, in SYMMETRIX™ B, which is the remote mirror to the 
source volume Rl in SYMMETRIX™ A, would be made 

60 "not-ready"' to Host 2. Normally, writes to the target volume 
R2 on Host 2, occur via the link paths created by the remote 
link director. However, if the source volume Rl on SYM- 
METRIX™ A fails, SYMMETRIX™ A will transparently 
access the data on its corresponding target R2 volume (here, 

65 Ml in SYMMETRIX™ B) 

A target volume R2 typically has a default configuration 
mode of "read-only" to any host with access to its SYM- 
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METRIX™ unit, in this example, HOST 2. To enable SUMMARY OF THE INVENTION 

"'T^ '^''^if'^-.u^ 'It^^' These and other objects are achieved by a host system for 

""''^ ^"'^ "^'^o^' f^"^^' remote control of mass storage volumes tising cascading 

TRIX™ unit contaimng the target volumes R2. As data is commands which collect information indirectly about a 

written to a sour^ volumeRl by HOST 1, the remote hnk 5 jj^^^ ^^j^^^^ ^^^^^^^ ^^ ^ ^^j^^^^ 

director RLD in SYMMETRIX™ A automatically wntes a ^^^^^ physically separate sites. 

COPY of that data over hnk path RAO or the SRDF connec- * i_ * * • j- j 

vjyy aiv4aiav^ p ^ , . , A host computcr program issucs the cascadmg commands 

tion to the corresponding target volume R2 on the designated „,u--,u T ^f^,™ 4^ 

o^^»*»*T-rr^TAr vT . r 1 r -1 which ask the locaUy commumcating mass storage system to 

SYMMETRIXB system. Thus, if a source volume Rl fails , -r l-l i. 

o X i 11 1 ivxy^ ojiM^iii i ju o, 11 a ^ X return information which can be used to identify one or more 

for any reason, the remote link dnrector RLD will transpar- lo ^^^^^^ ^^^^^^ ^ ^^^^^ ^ 

ently access the data on the corre^ding target volume R2 ^ ^ j^^^ j ^^^^ identified, commands can 

on the designated target SYMMETRIX B system over the . -1 ^ . . . ..^ .u-^^^u i u,. «^ ^>««- 

t- , J . . . ^ oe sent by the host through the locally commumcatmg mass 

link paths RAO and transmit that to HOST 1. . *• ^ 

iiuA paiuo xx^v aiju uauaiiiii luai i±v^v:>i a. storagc systcm to catise actions to occur at the identified 

New writes to the failed source volume Rl in SYMME- remote level, whether or not there are multiple intervening 
TRIX™ A accumulate as invalid tracks in the cache of 15 j^^^j^ remote mass storage systems. In the embodiment 

SYMMETRIX^M A— the unit containing the source volume shown, a host computer can query, establish, split, 

Rl. When the failing source volume Rl is replaced, the re-establish, copy and restore business continuance volumes 

remotely mirrored pair is re-synchronized automatically as a at any level in a chain of local and remote mass storage 

background operation within the SYMMETRIX™ unit, system sites. Those skilled in the art will appreciate that 
using the data in the appropriate target volume R2. 20 ^^^^^ commands could be implemented as well to effect 

FIG. 8 (Prior Art) shows an extended distance implemen- changes in the volumes at remote sites, 

tation of the SRDF features. So far, the above discussion it is an aspect of the present invention that it enables host 

dealt primarily with an SRDF CAMPUS connection, which commands to cascade over one or more intervening levels of 

usually involves physical proximity, such as units located in mass storage systems at remote sites to take effect at a 
two buildings on a college campus. In extended distance ^ particular designated site. 

configuration shown in FIG. 8 (PriorArt), the source volume gj^n another aspect of the present invention is that it 

Rl on SYMMETRIX^ unit SYMMETRIX A is being ^^^^^^^ ^ ^^^^ ^^^^^^^ operations being controlled at 

automatically and transparently copied to target volume R2 designated remote site. 

on SYMMETRIX C over the SRDF EXTENDED DIS- ^. n *u * f *u * • ♦ • *u » t 
T-AMi-c *• Tn, cDrMr- cv-rcxTT-icr^ T-Mc^AKTi-c 30 Still another aspect of the present invention is that it can 

TANCE connection. The SRDF EXTENDED DISTANCE . . * . , i^m-. * i_ ir 

, , . . . .11 significantly improve the remote site s abiuty to be effective 

solution allows source volume Rl to be automatically cop- u • *• •* j- * 

. , , . , , „^ ovx>rx>fT-'T-r»Tv u u u ^s a busmess continuance site for disaster recovery, 
led to target volume R2 on SYMMETRIX C, which may be 

located thousands of miles away. Thus, if a disaster such as BRIEF DESCRIPTION OF THE DRAWINGS 

a flood destroys all the hosts and SYMMETRIX™ units ^ • , ^. 

located at a campus site, the application can still be run from FIG. 1 is a block diagram of the present mvention. 

the remote, extended distance site represented here by FIG. 2 is a block diagram illustrating establishment of a 

SYMMETRIX C. Sites can be as much as 37.5 miles (60 local business continuance volume. 

km) apart from each other in a "campus" solution, or in an FIG. 3a is a block diagram illustrating use of the present 

extended distance solution over 37.5 miles (60 km) apart invention with two linked sites and the establishment of a 

using T3 or E3 or similar high speed links. business continuance volume at a remote site. 

These disaster recovery remote sites, whether campus or FIG. 36 is a block diagram illustrating another use of the 

extended ones, are mirrors or copies of data stored on the present invention. 

SYMMETRIX™ systems physically located near the host FIGS. 4a-4e block diagrams of the syntax of cascading 

computer. As described above, the copying is done by the remote business continuance volume commands of the 

respective SYMMETRIX™ systems automatically, once all present invention. 

appropriate SYMMETRIX^^" systems have been configured piG. 5 is a detailed view of cascading business continu- 

for the SRDF feature. This means the host(s) and local ancc volume command of the present invention. 

SYMMETRIX™ systems at each site are able to operate as ^ ^^^^^ diagram of an establish command using 

efBciently as if the remote copying were not occurnng. That cascading commands of the present invention, 

is, the continuous copying to the remote sites IS not "logi- - i.i i c j i i * j 

11 - -t-i « * iT * * * *i- • '* FIG. 6o IS a block diagram of a differential split command 

cally visible to the host computers at the vanous sites, nor ..u ^ c .x, * ■ 

^ ' ^ r • * u * « u * using the cascading commands of the present mvention. 

does It mterfere with processmg at the host. Hence, a host ^ , ^ 

computer, is not able to send commands that effect changes ^ ^^^^^ diagram of a reverse split command 

to the remote sites. This is true even if the remote sites are "^ing the cascading commands of the present mvenUon. 

located only one step away in a campus solution. If the host FIG. 6d is sl block diagram of a reverse differential split 

wishes to effect changes, either the CAMPUS or the command using the cascading commands of the present 

extended sites, there was heretofore no way of doing this invention. 

directly at the local site. FIG. 7a (PriorArt) is a block diagram of the physical and 
It is an object of this invention to enable a host computer ^ logical disk structures of an HSM system. 

to cause volumes to be managed remotely. FIG. 76 (PriorArt) is a block diagram of disk mirroring. 

It is another object of the present invention to enable a FIG. 7c (Prior Art) is a block diagram of disaster recovery 

host computer to issue commands to effect changes in features linking two physically separate sites. 

volumes at remote SRDF sites. FIG. 7d (Prior Art) is a block diagram of remote disk 
Still another object of the present invention is to enable a 65 mirroring using SYMMETRIX™ SRDF features. 

host computer to collect information about volumes at FIG. 8 (Prior Art) is a block diagram of an extended 

remote SRDF sites. distance configuration of a SYMMETRIX™ system. 
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FIG. 9 is a block diagram of the logical flow of data 
between a host operating system and mass storage sub- 
systems. 

FIG. 10 is a block diagram of simple mirroring in a mass 
storage subsystem. 

FIG. 11 is a block diagram of a mass storage subsystem 
in which a local business continuance volume is being 
established. 

FIG. 12a is a block diagram of a mass storage subsystem 
in which a local business continuance volume has been 
established. 

FIG. 12b is a block diagram of a mass storage subsystem 
in which a local business continuance volume has been split 
from its standard volume. 

FIG, 13, is a block diagram of the re-establishment of a 
business continuance volume in a local system. 

FIG. 14 is a block diagram of a restore command taking 
effect in a BCV volume in a local system. 

FIG. 15 is a flow diagram of the present invention. 

FIG. 16 is an illustrative table used with the present 
invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

In FIG. 1, a block diagram of the present invention is 
shown. In this example, a host computer system, host 02, is 
executing an operating system MVS, which is IBM Corpo- 
ration's MVS operating system. Those skilled in the art 
appreciate other operating systems for a host, such as UNIX 
or MICROSOFT'S NT™ operating systems could be used, 
among others. Host component software 08 allows a user to 
cascade commands from host 02 to a SYMMETRIX''^^ 
system such as SYMM2, at the LOCAL site, to SYMME- 
TRIX™ SYMM5 at the CAMPUS site to SYMMETRIX™ 
system SYMM6 at an EXTENDED site. 

Still in FIG. 1, commands can be issued from host 02, 
SYMM2 to take effect at SYMM5 and SYMM6, for 
example, by establishing a BCV volume (RIBCV) in 
SYMM5, which causes the contents of standard volume R2 
in SYMM5 (which are actually the contents of RIB in 
SYMM2) to be copied to RIBCV. When the RIBCV volume 
is subsequently split off, it becomes a source volume 
RIBCV in SYMM5 to target volume R2 in SYMM6. Thus, 
the contents of disk RIB, in SYMM2 at the LOCAL site, 
have been propagated to disk R2 in SYMM6 at an 
EXTENDED DISTANCE location possibly thousands of 
miles away. All of the copying needed to effect this is 
transparent to each of the hosts at each site. In this example, 
the contents of disk R2 on SYMM6 at the EXTENDED 
location can serve as a disaster recovery backup, as of a 
certain time and dale, for the business applications running 
on host 02. 

Turning briefly to FIG. 9, an overview of the interface 
between host software 08 and a SYMMETRIX''^" system 
SYMMl is shown. Host 02 is connected to mass storage 
subsystem SYMMl. In the MVS environment, the operating 
system software MVS links computer programs to physical 
devices by means of a unit control block, UCB, which, in 
turn, maps into the MVS operating system's channel sub- 
system. The channel subsystem takes logical input/output 
requests and maps them into channel path selections to a 
particular device and ultimately, physical input/output sig- 
nals for that device. The logic inside a SYMMETRIX™ 
system does the final mapping which assigns the logical 
device to one or more mirrors and identifies the SYMME- 
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TRIX™ device numbers for each device and mirror 
involved, in this case, DA1C1H2, and DA1C2H3 and 
DA1C4 HI. Those skilled in the art will appreciate that other 
operating systems have similar mechanisms for separating 
5 the logical input/output requirements from the physical as 
this enables much more flexibility for configuring the hard- 
ware. 

U.S. patent apphcation Ser. No. 08/842,953 filed Apr. 
25^, 1997, issued as U.S. Pat. No. 6,101,497 on Aug. 8*^, 

10 2000, entitled "Method and Apparatus for Independent and 
Simultaneous Access to a Common Data Set", ("953") 
which assigned to the Assignee of the subject application, 
and is also hereby incorporated by reference, generally 
describes a way to further enhance SYMMETRIX™ sys- 

15 tems by allowing the establishment of what are known as 
Business Continuance Volumes (BCV) in a SYMME- 
TRIX™ system. In addition, co-pending U.S. patent appli- 
cation Ser. No. 09/039/020, filed Mar. 13, 1998, entiUed 
"Host System for Mass Storage Business Continuance Vol- 

20 umes" ("020^') which is assigned to the Assignee of the 
subject application and is also hereby incorporated by 
reference, generally describes a system for enabling a host 
computer, using an operating system such as MVS, to find 
the time to establish and manage such business continuance 

25 volumes in a local SYMMETRIX™ system without dis- 
rupting critical business programs. 

Since the establishment of Business Continuance Vol- 
umes (BCV), as generally described in application 020, 
makes use of ordinary mirroring capabiHties found in SYM- 
METRIX™ units and similar HSM systems, simple mirror- 
ing is illustrated at FIG. 10. 

In FIG. 10, the logical volumes as the host sees them are 
shown in the box labeled HO. In this and the next few 
Figures, the host recognizes two logical volumes by the 
volume serial numbers VoLA and VolB. Mass storage sub- 
system SYMMl actually stores the data for these volumes 
on its own logical volumes. Here, mirroing has been con- 
figured for the data on a logical SYMMETRIX™ volume 
with a volume serial number of VoLA. Mass storage sub- 
system SYMMl assigns this logical volume to logical 
volume 20aml. To create a mirror disk for it, it assigns 
another logical volume 20am2. Mass storage subsystem 
SYMMl maintains logical volumes 20aml and 20am2 as 
mirror copies of each other, but host HO and the MVS 
operating system work with logical volume serial VolA as 
though it were the only logical volume in use. 

Next, as generally described in co-pending application 
020, SYMMl uses the same mirroring techniques to estab- 

50 lish a business continuance volume BCV 30b. Any mirror 
structure can be used, such as normal mirroring, RAID, 
mirroring with the Assignee's Symmetrix Remote Data 
Facilities (SRDF), and so on. 

The establishment of a BCV volume, using the ESTAB- 

55 LISH BCV command is shown in FIG. 11. Host component 
software causes mass storage subsystem SYMMl to set 
logical volume 306 as "not ready" to host HO (and any other 
hosts H connected to mass storage subsystem SYMMl.) 
Mass storage subsystem SYMMl then assigns BCV volume 

60 30b as the next available mirror for standard volume mirrors 
20aml and 20am2. This assignment of the BCV volume to 
mirror status enables mass storage subsystem SYMMl to 
copy the contents of standard mirror volumes 20aml and 
20am2 to BCV volume 30b (20am3) as shown by the arrow 

65 C in FIG. 11. While the copying is taking place, it is 
transparent to the host, which may continue to make I/O 
requests to logical volume VolA. For better performance. 
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half the data is copied to BCV volume 30b from disk 20aml configuration. Typically, LOCAL connections are limited to 

and the other half from disk 20am2. a specified area defined by cable capacities which are 

Now as is shown in FIG. 12a, when the copying is uaiaUy equivalent to the confines of one building or floor of 

complete and BCV volume 30fc(20am3)'s contents arc iden- * buildmg. 

tical to those of standard volume lOaml and 20am2, BCV 5 Usmg the Assignee's SRDF remote disaster recovery 

volume 30b (20am3) is now synchronized with standard S^l^^f/^^"*^? f "T!' %pni SYMMETRIX™ system 

volumes 2»aml and 20a«i2 and capable of working as a ^YMMl can be Imkedby SRDF as shown in FIG. 1, to a 

J TM_ - * physically separate SYMMETRIX^** system located else- 

business continuance device. This copying is transparent to, ^^^^ CAMPUS. As mentioned above, CAMPUS 

and notdisrupuve ofthebusine^apphcationnimiingonthe ^ ^^^^ ^.^^ ^^^^ 

host. Once a BCV volume has been synchronized by mass ^vAifxyrcTOivTM if tu^ ™,«« «,«oo 

r.xrx*x*^ -.- -J J .r T%/->x7 SYMMETRIX™ system. If the company using the mass 

storage subsystem SYMMl it is considered part of a BCV ^ ^ ^^^ and disaster recovery feamres of the SYM- 

pair and any new data written to the standard volume of ^ETOIX™ system is a nationwide or international 

such a pair is also wntun to the BCV volume rf^^^ BCV J ^ SYMMETRIX™ system SYMM5 at 

pair by mass storage sutey^tem SYMMl. The BCV volume cAMPUS can also be linked to a SYMMETRIX™ system 
wiU continue to be marked as not ready for any hosts H 15 ^^^^ ^ EXTENDED location, which may be thou- 

which use mass storage subsystem SYMMl as long as it is ^^^^ ^y^^ ^^^^ 

a part of a BCV pair. , . . , . SYMMETRIX™ systems in CAMPUS and EXTENDED 

FIG. 126 shows the result of a spht command described ^cations are able to communicate with each other when the 

in apphcation 020. The spht command allows the user to g^pp jj^ jhs such as remote access groups RAO, RAl 

stop the mirroring operation and frees up the new copy of the j^j, ^^^^ j,^^^ estabhshed by EMC Corporation's 

data for other purposes, such as testong or restore operauons. g^pp features. However, heretofore, a host 02, was only 

The response to a split command suspends any I/O to the ^^j^ ^^ communicate with SYMMETRIX™ systems near it 

BCV vohimc 30fc of the BCV pair, changes its device state ^ LOCAL. Commands that would have previously 

to ready, enables access to it from any host H, while stiU ^nly been possible to send to the LOCAL SYMMl and 

containing the volume serial number which is the duplicate sYMM2 shown in HO. 1, can now be sent to remote devices 

of the standard volume.The device is still offline, after the ^^ sYMMETRIX™ systems SYMM5 and SYMM6 at 

split since it was marked offline in order to originally cAMPUS and EXTENDED locations, using the present 

estabUsh or restore the volume. Mass subsystem SYMMl invention 

flags any new writes to the standard volume that occur after ^ ^j^^ Corporation's SRDF facility has linked 

the split (m case the BCV is re-estabhshed later) and ^^^^ ^^MMl and SYMM2 to remote sites at 

resumes normal operations. New wntes to the BCV volume campuses or at extended locations, but again, with the goal 

are also flaeeed, after the spht so that when a re-establish or a - n *u *• •* * « * u ♦ u u ^* 

T • . . . 1 - J -1 - of keeping all the activity transparent to hosts such as host 

mcremental restore is done, as descnbed m more detail in ^^^t- c.„^*«,^ xA/c u^^t oj err- i 

. . L . o^z**ik>ri 02- Inus, operating system MVS in host vZ oi rlO. 1, 

application 020, mass storage subsystem SYMMl can com- i^^i^^ing the SYMMETRIX™ systems host component 

pare the changes to both volumes and refresh the tracks that ^^^^^^^^ ^^^^^ ^^j^ ^^^^ ^^^^ ^^^^^ immediate 

have changed. SRDF status in its vicinity, but was unable to effect changes 

FIG. 13 shows the process of re-estabhshmg a BCV pair, gp^pp volumes and was unaware of the status of any 
in which only the changes that have been made to standard gj^^p devices downstream in the stream of linked SRDF 
volumes 20aml and 20am2 are copied to volume 30b. Any devices and unable to effect any changes in them. In the 
tracks that had been changed on volume 30b as a result of present invention, this is changed, so that the status of 
its use after a split, are overwritten by the data on the devices at remote CAMPUS or EXTENDED sites, or any- 
corresponding track on the standard volumes. ^^^^^ stream can be determined by host 02's oper- 

FIG. 14 illustrates the restore command which restores ating system using the SYMMETRIX™ systems host com- 
standard volumes 20aml and 20am2 with all the data from 45 ponent software 08 of the present invention, and commands 

the BCV volume, now seen as volume 30b(20am3y A can be sent downstream to take effect one or two or more 

restore is done by setting the BCV volume as the next SRDF locations away. In effect, as seen in FIG. 1, with the 

availablemirrorof the standard device, and copying its data present invention, commands can "skip over" or "pass 

to both volumes 20aml and 20am2, overwriting the data through" a locally attached SYMMETRIX™ system, and a 
present on those devices. 50 CAMPUS SYMMETRIX™ system, to take effect at an 

Returning to FIG. 1, when EMC Corporation's SYMME- EXTENDED SYMMETRIX™ system. 
TRIX™ systems, such as SYMMl and SYMM2, are Turning briefly now to FIG. 5, an illustrative command 
installed locally and connected to a host, they, in effect, for collecting and managing information remotely is shown, 
replace the simple physical devices known to the operating in this example, a Query command 50 is issued by host 
system MVS as MVS devices identified by an MVS device 55 component software 08 to MVS device cuu. The sequence 
number cuu, which identifies an MVS device on a particular number parameter functions as described in the above- 
controller, with sophisticated HSM mirroring and redun- referenced co-pending 020 patent application (a sequence 
dancy systems. Thus, when the user is sending commands to number can be a decimal number from 1 to 128 which 
MVS device number cuu, SYMMETRIX™ host component indicates in what order the specified command is to be 
software 08 actually sends physical input and output signals 60 executed. All actions on the same sequence level are 
through the operating system MVS's channel subsystem to executed in parallel.) The remote features of this command 
the SYMMETRIX™ systems. In this example, MVS device are shown at RMT 54, cuu 56 and ragrp 58. RMT 54 
number 101 might be mapped to a SYMMETRIX'^^ device parameter specifies that the query is to gather information on 
on SYMMl, which identifies a disk adapter DA and a the remote devices in an SRDF disaster recovery configu- 
physical disk C. 65 ration. Ragrp 58 specifies the RA group through which the 

As seen in FIG. 1, host 02 and SYMMETRIX'''^ systems user wishes to perform a remote operation. For a remote 
SYMMl and SYMM2 are physically together in a LOCAL command, where a remote access group (ragrp), has not 
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been specified, the cuu 56 parameter is interpreted by user to get host component software 08 to determine the 

SYMMETRIX™ systems to refer to source devices (known status of any BCV volume) with remote parameters (as 

as Rl devices) for SRDF functions. Where a remote access shown in FIG. 5 and described above) to locate all or a 

group ragrp 58 is known, this access group can be specified. specified number of the BCV volumes at the remote sites 
As seen here in FIG. 5, by the syntax of the command, the 5 related to the logical volume in question. Since the SRDF 

ragip 58 operand is optional and can be positional. Thus, if features in the SYMMETRIX™ systems already maintain 

the user wished to query the status of R2BCV volume in the information about the remote volumes inside the SYM- 

SYMM6, of HG. 1, such a query might appear as: METRIX™ systems themselves, in a table such as the 

Query 1,RMT (202,1, 0,ALL), where cuu is MVS device illustrative Table in FIG. 16, the remote parameters identify 
number 202 in SYMM2, 1 refers to ragrp RAl between lO sets of remotely linked devices the user wishes to know 

SYMM2 and SYMM5, and 0 refers to ragrp RAO between about. 

SYMM5 and SYMM6. FIG. 2 illustrates the way in which the BCV features 

Tbming briefly to FIG. 7c (Prior Art), source volumes Rl operate with an SRDF source volume Rl, on SYMMl using 

are the set of volumes labeled here as S1-S8. When the the present invention. As seen here, an establish BCV 

SRDFfeatm-eisactivated,aremotelinkdirectorRLD treats command issued for source device Rl, will result in the 

these volumes as a remote access group, here RAO. The establishment of a BCV volume on SYMMl of host 02. 

SRDF feamre links the source volumes Rl to target volumes NormaUy, the data for the BCV volume will be copied from 

R2 in SYMMETRIX B(SLAVE) shown here as volumes the source volume Rl on SYMMl, unless problems neces- 

T1-T8), under the heading of remote access group RAO, sitate obtaining the data from the target volume R2 device in 

Turing back to FIG. 1, it can be seen that SYMME- ^^e SYMM3 device attached to host 04. If the BCV device 

TRIX™ system SYMMl contains a source volume Rl is spht from the source device, Rl this does not affect daU 

which is linked to remote SYMMETRIX™ system transfers between source volume Rl and target volume M 

SYMM3, in the CAMPUS complex by SRDF over remote restore and mcremental restore commands, if the source 

access group RAO. SYMMETRIX™ system SYMM2, at ^^^^^^ Rl/target volume R2 logical device hnk is 
physical location LOCAL, contains two source volumes ^ suspended, the restore or incremental restore is only naade 

RIA and RIB, which are linked to SYMMETRIX™ sys> locally to the source Rl device and the information about 

terns SYMM4 and SYMM5 by SRDF over remote access ^'^^"S^^ tracks is retained for later synchronization with the 

groups RAO and RAl. Note that in this configuration, ^^^^et volume R2. 

SYMMETRIX™ system SYMM3 is referred to as remote Turning now to HGS. 3a and 36, the present invention 

access group 0 (RAO), in relation to SYMMl. SYMME- also enables BCV volume activities at remote locations to be 

TRIX™ system SYMM4 is referred to as remote access controlled through cascading commands issued from a host 

group 0 (RAO), in relation to SYMM2, while SYMME- ©2. Establish and reestablish processes are the same as those 

TRIX™ system SYMM5 is referred to as RAl, in relation described in application 020 if BCV devices are used in 
to SYMM2. This latter SYMMETRIX™ system SYMM5, 3^ conjunction with an target volume R2 as the standard device, 

is, in turn, linked by SRDF over remote access group RAO Turning now to FIG. 3fl, the present invention allows a 

to a SYMMETRIX™ system SYMM6 at an EXTENDED user at host 02 to establish a business continuance voliune 

location. SYMMETRIX™ system SYMM6 is therefore a BCV at a remote CAMPUS location, linked by SRDF to 

remote access group RAO to SYMMETRIX™ system SYMMETRIX™ system SYMM3. In this approach, the 
SYMM5. The configuration shown in FIG. 1 is for illustra- ^ SRDF functions have already linked source volume Rl of 

tive purposes only. Other configurations are possible. SYMMl to target volume R2 of SYMM3, by assigning it as 

Still in FIG. 1, by use of the present invention as well as a remote mirror, thus assuring transparent copying of the 

the BCV features described in co-pending applications 953 contents of Rl to R2. As noted above, in the discussion of 

and 020, the mass storage volume SYMMETRIX™ system the SRDF feaUires, this copying is done in the background 
SYMM6 contains remotely located copies of the data on 45 by the SRDF features of SYMM3 and does not take time 

source volume RIB in SYMMETRIX™ system SYMM2. away from normal processing at host 02. Similarly, the 

The copies residing on SYMMETRIX™ system SYMM6 copying done to create a BCV volume is also done in the 

may actually have been made from the intervening level background in SYMM3, using the BCV features generally 

source volume RIBCV in SYMMETRIX™ system described in co-pending apphcations 953 and 020, without 
SYMM5. When SRDF is used to link SYMMETRIX™ 50 taking time away from normal processing in host 02 or host 
systems, each system knows its place in the chain. Thus, the 

present invention enables a user to establish a remote BCV The unique aspect shown in FIG. 3a is that the user can 

volume, RIBCV in SYMM5. Heretofore, a user could only intervene at a remote SYMMETRIX™ system SYMM3, 

establish a BCV volume on a locally configured SYMME- using the cascading commands of the present invention, to 
TRIX system. Once the remote BCV volume has been 55 establish a BCV volume there. If a split command is 

established, it becomes a mirror of the target volume R2, on subsequently done, the SYMMETRIX™ unit containing the 

SYMM5, which, in turn, by virtue of the SRDF fiinctions, target volume R2 device associated with the BCV pair locks 

was already a mirror of the source volume RIB in SYMM2. the target volume from further updates and suspends the 

If the user wishes to make yet another remote copy, the user remotely mirrored Rl and R2 pair for a few seconds to 

uses the remote split command to split RIBCV off from execute the split. When the split is complete the link is 

target volume R2 in SYMM5, makes RIBCV a source restored, any changed tracks from source volume Rl are 

volume Rl, and, using SRDF, links it to target volume R2 in propagated to the target volume R2 for synchronization 

SYMM6. Thus, the data which was copied to RIBCV from purposes. In this example, after the split, a BCV device 

Rl in SYMM2 can now be cascaded down to target volume containing a copy of the standard taiget volume R2 data is 
R2 in SYMM6. 65 now available for use by host 04. 

With the present invention, host component software 08 In FIG. 3^?, another aspect of the present invention is 

can issue a query command (a query command allows the shown. Since the SRDF features of SYMMETRIX™ sys- 
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tems can also be bidirectional, as well as unidirectional, 
SYMMl can be configured by SRDF functions to contain a 
target volume R2, which is a remote mirror of source volume 
Rl in SYMM3. As host 04 updates source volume Rl, in 
SYMM3 its contents are automatically copied by SRDF 
features to target volume R2 in SYMMl. In this example, 
the present invention enables a user at host 02, to establish 
a BCV volume in SYMMl from target volume R2 in 
SYMMl. As mentioned above, a split command can then be 
used to stop the copying operation from R2 to the BCV 
volume, and frees up the new copy of the data on the BCV 
volume for other purposes, such as testing or restore opera- 
tions. Effecting a spUt command does not affect SRDF data 
transfers with the target R2 device. 

Now referring to FIG. 6a, an establish with remote 
mirrors is shown. In FIG. 6a, data volumes DVOO and DVOl 
are depicted as they are seen by the host. When these are 
established as a BCV pair, source volume BCVDVOl Rl is 
assigned as the next mirror (M3) to standard volume SDVOO 
and data from SDVOO Ml and SDVOO M2 is copied to 
BCVDVOl Rl, synchronizing it. Since, in this example, 
BCVDVOl Rl is also a source volume in an SRDF link, it 
will be synchronized with target volume BCVDVOl R2. In 
many cases, a BCV pair may be split and re-established a 
number of times. 

The present invention provides that the user may request 
a differential split, as shown in FIG. 6b, which sphts off 
BCVDVOl Rl from the standard volumes, and enables 
copying of only the updated tracks on BCVDVOl Rl to its 
remote mirror BCVDVOl R2. 

FIG. 6c illustrates a reverse split. A reverse split resyn- 
chronizes the BCV volume, here BCVDVOl Rl with a full 
data copy from its mirror — here remote target volume 
BCVDVOl R2. Prior to the present invention, without a 
reverse spht command, a user could not put a BCV volume 
back to its original state if there had been any writes to the 
standard volume while the BCV volume was attached. When 
doing a restore, from a BCV device to a standard device no 
I/O commands could be sent to the standard device, since 
they would be propagated to the BCV device, thus destroy- 
ing its integrity before the restore was complete. Heretofore, 
if a user wanted to preserve the data contents of the BCV 
device, the user had to wait for a restore to complete, and 
then issue a split command. When the restore completed, 
I/O's to the standard device could resume. Since a restore 
might take anywhere from 1-15 or more minutes, or might 
not be scheduled for hours, keeping the integrity of the data 
contents of the BCV device could be costly for performance 
of the system using the standard device. Yet, frequently, 
BCV devices are used to capture the status of data as of a 
certain day and time, so testing can be done against this copy 
of the real data. If the systems using the test data have bugs» 
it is often easier to find them by using the same data for 
testing until all bugs are fixed. Thus, for some applications, 
maintaining the integrity of the BCV data is important. 

With the reverse split of the present invention, as soon as 
a reverse spht is initiated, a user can immediately start using 
the standard device of the BCV pair, and send I/O operations 
to it. This is because the copying in a reverse split goes from 
the BCV device's mirror, remote mirror BCVDVOl R2 in 
this example, to the BCV device BCVDVOl Rl, thus 
keeping the BCV data intact. In a reverse split, a full copy 
of the data is transmitted from the remote (in this example) 
mirror. 

Turning now to FIG. 6dy a reverse differential split is 
illustrated. Here, only the out-of-sync tracks are copied 
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(instead of a full copy) from remote mirror BCVDVOl R2 
(in this example) to the BCV device BCVDVOl. 

Thus, the use of the reverse split and reverse differential 
split allows a user to keep the BCV data contents as of a 
certain date and time unchanged on the remote mirror (in 
these examples), and use them to return a changed BCV 
volume to the original contents of that date and time. All of 
this can be done without interrupting processing on the 
standard volumes. 

Returning now to FIG. 1, it can be seen that if restore 
operations were the only way to restore BCV data from a 
remote site, such as SYMM6, the delay could be further 
compounded by telecommunications delays over the SRDF 
EXTENDED DISTANCE link. With the reverse differential 
split, a changed BCV volume can be brought to the status of 
the BCV data as of a certain date with no interruption of I/O 
to the standard devices. 

Still in FIG. 1, SYMMETRIX^m systems capable of 
implementing the cascading commands of the present inven- 
tion include microcode in the remote link directors and other 
parts of the SYMMETRIX™ system, which gives a host 
attached to the SYMMETRIX^" system at a source location, 
the ability to issue cascading BCV commands across SRDF 
links to devices in the SYMMETRIX™ at target locations. 
These cascading BCV commands are pass-through com- 
mands that travel across a link defined by an RLD as an 
R1/R2 pair, such as that shown in SYMMl and SYMM2. 
The commands can also be passed through a configured 
socket device that is a member of the specified remote access 
(RA) group. In FIG. 1, this is exemplified at SYMM5 and 
SYMM6, where SRDF communications occur over a tele- 
communications link RAO. In FIG. 16, the microcode tables 
kept by each SRDF configured SYMMETRIX™ system are 
modified in accord with the present invention to include an 
additional address indicator — ^linel, addr — for indicating the 
place of this unit in a chain. This address indicator can be as 
simple as additional bytes to include more references, or as 
complex as a vector which points to other addresses. 

Turning now to FIG. 15, a flow diagram of the present 
invention is shown. Since most of the processing for BCV 
volume commands, such as establish, split, query, 
re-establish, restore, etc., are diagrammed in application 
020, referenced above, this flow diagram concentrates on the 
general flow followed for parsing and acting upon remote, 
cascading commands — block 150. At decision block 155, 
the present invention checks the command to see whether 
the RMT parameter has been specified. If it has not, local 
processing continues at step 160. If it has been specified, the 
present invention parses the command syntax (illustrated in 
FIGS. 4a~4e and FIG. 5) to identify the remote access group 
ragrp and remote devices which are the targets of the 
command. As noted above in the discussion of FIG. 5, this 
command syntax uses a simple positional remote access 
group number to identify the path. The invention checks at 
step 175 to see if an ragrp has been specified. If it has not, 
at step 180 the system checks to see if the device specified 
is a source volume, Rl. If it is not, then this is an error in a 
remote command. If the device is an Rl, the invention 
proceeds to step 200, and uses the ragrp number that the Rl 
is associated with, and proceeds to step 170. If an ragrp was 
specified or has been assigned as in step 200, at step 170, a 
path is defined and the pointer to the first ragrp link in it is 
indicated. At that point, the present invention returns to 
normal command processing. When a remote cascading 
command is issued, it follows the link paths defined for it by 
the user, and the present invention. Thus, a command can 
take effect several "hops" away from the local system. 
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While the present invention applies these cascading com- 
mands to BCV commands, those skilled in the art will 
appreciate that the same apparatus and method could be used 
for other commands designed to take effect at a distance 
through several intervening systems. 

Those skilled in the art will appreciate that while a 
preferred embodiment of the present invention is directed to 
handling mass storage disk volumes connected to host 
systems, other types of writable storage media requiring 
remote access could also be handled by the method and 
apparatus of the present invention. Similarly, while the 
remote commands of the present system are illustrated 
primarily for use with business continuance volumes, those 
skilled in the art will appreciate that they could apply to 
other types of activity in a remote system setup. 

In this embodiment, host component software OS's imple- 
mentation of the remote control cascading commands is 
written in assembler language for IBM mainframes using the 
MVS operating system. Those skilled in the art will appre- 
ciate that it could also be written in other languages, such as 
C or C++, for example. Similarly, while the present inven- 
tion is designed for use with the MVS operating system, 
those skilled in the art will appreciate that variations of it 
could be implemented for other operating systems such as 
UNIX™, NT™, and others without deviating from the spirit 
of the present invention. 

Those skilled in the art will appreciate that the embodi- 
ments described above are illustrative only, and that other 
systems in the spirit of the teachings herein fall within the 
scope of the invention. 

What is claimed is: 

1. In a system comprising a local mass storage system 
which is linked to a host system and to a physically remote 
mass storage system, the physically remote mass storage 
system being linked to one or more other physically remote 
mass storage systems so as to form a stream of mass storage 
systems, each pair of mass storage systems being linked by 
a communications path and a pair of remote Hnk directors, 
each of the mass storage systems comprising one of the pair 
of remote link directors, and each such mass storage system 
further comprising a cache memory, a disk adapter and one 
or more disk drives, a host system providing local control of 
mass storage volumes located on the local mass storage 
system and remote control of linked mass storage volumes 
located on any of the remote mass storage systems in the 
stream of mass storage systems, comprising: 

a mechanism at the host system that generates and issues 
a user-specified command from the host system to a 
local mass storage system; 
a selected identifier being in the command as issued from 
the host system, the command being recognizable by 
each mass storage system in the stream of mass storage 
systems, the selected identifier being selected from a 
group of identifiers, the identifiers in the group identi- 
fying the mass storage systems in the stream of mass 
storage systems, each mass storage system that is not 
identified by the selected identifier being configured to 
forward the command to another mass storage system, 
at least one mass storage system that is identified by the 
selected identifier being configured to execute the com- 
mand after the at least one mass storage system 
receives the command, 
a plurality of user-specifiable parameters also being speci- 
fiable in the command as issued from the host system, 
the parameters that may be specified in the command 
including a first parameter, a second parameter, and one 



or more third parameters, the presence of the first 
parameter in the command indicating that the command 
is to be executed physically remotely from the local 
mass storage system, the second parameter indicating 

5 an order in which the command is to be executed, and 
the one or more third parameters indicating one or more 
paths via which the command is to be forwarded to the 
at least one mass storage system, the absence of the first 
parameter firom the command indicating that the com- 

10 mand is to be executed at the local mass storage system. 

2. The host system of claim 1, further comprising host 
component software for processing command syntax to 
allow a user to specify business continuance operations to be 
effected on the at least one mass storage system that is 

15 identified by the selected identifier. 

3. The host system of claim 2, wherein said business 
continuance operations comprise an establish command for 
causing said at least one mass storage system that is iden- 
tified by the selected identifier to join a specified standard 

20 volume at a local site and a specified business continuance 
source volume into a business continuance pair by copying 
contents of said standard volume onto said business con- 
tinuance source volume and a target volume mirror at the at 
least one mass storage system. 

4. The host system of claim 3, wherein said business 
continuance operations further comprise a split command 
for causing said at least one mass storage system to separate 
said business continuance pair and flag a write to said 
standard volume and said business continuance source vol- 
ume which may occur after said split. 

5. The host system of claim 4, wherein said business 
continuance operations further comprise a reverse split 
command for causing said at least one mass storage system 
to separate said business continuance pair and copy all the 
contents from the business continuance volume to the asso- 
ciated source volume. 

6. The host system of claim 4, wherein said business 
continuance operations further comprise a reverse differen- 
tial split command for causing said at least one mass storage 

40 system to separate said business continuance pair and copy 
only changed contents from the business continuance target 
volume to the associated source volume. 

7. The host system of claim 2, wherein said business 
continuance operations comprise a re-establish command 

45 for causing said at least one mass storage system to rejoin a 
standard volume and a business continuance source volume 
which had once been a business continuance pair into 
another business continuance pair, by copying to said busi- 
ness continuance source volume writes to said standard 
volume which had been flagged after a split and copying 
from said standard volume any data which had been changed 
by writes to said business continuance source volume which 
had been flagged after said split. 

8. The host system of claim 2, wherein said business 
continuance operations comprise a restore command for 
causing said at least one mass storage system to copy 
contents of a business continuance source volume of a 
business continuance pair to a standard volume. 

9. The host system of claim 2, wherein said business 
60 continuance operations comprise an incremental restore 

command for causing said at least one mass storage system 
to copy to a standard volume writes to a business continu- 
ance source volume which have been flagged since said 
business continuance source volume was split from a busi- 
65 ness continuance pair. 

10. The host system of claim 2, wherein said business 
continuance operations comprise a differential split com- 
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mand for causing said at least one mass storage system to lish command for causing said at least one mass storage 

separate a business continuance pair and copy only changed system that is identified by the selected identifier to join a 

data from the business continuance source volume to a target specified standard volume at a local site and a specified 

volume mirror. business continuance source volume into a business con- 

11. In a system comprising a local mass storage system 5 tinuance pair by copying contents of said standard volume 
which is Hnked to a host system and to a physically remote o^^^ business continuance source volume and a target 
mass storage system, the physically remote mass storage volumemirror. 

system being linked to one or more other physically remote \1- ^h*^ '"^thod of claim 13, wherem said busmess 

mass storage systems so as to form a stream of mass storage conUnuance operaUons fiirther comprise the step of usmg a 
systems, each pair of mass storage systems being linked by lO ^P^^^ command for causing said at least one mass storage 

,t_ J - i ^iTj- * system to separate said busmess contmuance pair and nag a 

a communications path and a pair of remote hnk directors, ^y^^y"^ o^|.«iaiv oax« t^o^i^w^ u f " & 

^o^K iu^ wv^^cc otJTr^^^ c.r^i^Ji.^ ^^,^«,;c-,-r,rr .^f iU^ wttte to said standard volume and said busmess contmuance 

each oi the mass storage systems comprismg one oi tne pair . _ 

r * j u u * * source volume which may occur after said sput. 

of remote link directors, and each such mass storage system _r . , 7^%. t. - .jt_- 

further c»mprising a cache memory, a disk adapter and one ^f. ^ "^t**"? °f ^^^f » f"*' ^"^^ 
or more disk drives, a method of enabling a host system to is conUnuance operaUons further compr^e the step of usmg a 

perform local control of mass storage volumes located on ^establish command for cauang said at feast one mass 

the local mass storage system and remote control of linked ^ " ^^"^"t volume and a business 

1 1 . J ^ ♦u^ ^^^^4^ continuance source volume which had once been a business 

mass storage volumes located on any of the remote mass . . . , . . . - . •_ 

^#«™ «f ™™ o,,o««,v«o continuance pair into the busmess continuance pair agam, by 

storage systems m the stream oi mass storage systems . • , *j 

comprising the steps of* copymg to said business continuance source volume said 

P ^ \ ^ , . writes to said standard volume which had been flagged after 

issuing a user-^eafied command from the host system to ^^^^ gp^^ ^^^j copying from said standard volume any data 

the local mass storage system; and ^^^^^ ^^^^ changed by said writes to said business 

placing at the host system a selected identifier in the continuance source volume which had been flagged after 

command, the command being recognizable by each said spHt. 

mass storage system in the stream of mass storage ^ method of claim 14, wherein said business 

systems, the selected identifier being selected from a continuance operations further comprise the step of using a 

group of identifiers, the identifiers in the group identi- restore command for causing said at least one mass storage 

fying the mass storage systems in the stream of mass system to copy contents of a business continuance source 

storage systems, each mass storage system that is not volume of a business continuance pair to a standard volume, 

identified by the selected identifier being configured to 17. The method of claim 14, wherein said business 

forward the command to another mass storage system, continuance operations further comprise the step of using an 

at least one mass storage device that is identified by the incremental restore command for causing said at least one 

selected identifier executing the command after the at mass storage system to copy to said standard volume said 

least one mass storage system receives the command, writes to said business continuance source volume which 

aplurality of user-specifiable parameters also being speci- have been flagged since said business continuance source 

fiable in the command as issued from the host system, volume was split from said business continuance pair, 

the parameters that may be specified in the command 18. TTie method of claim 14, wherein said business 

including a first parameter, a second parameter and one continuance operations further comprise the step of using a 

or more third parameters, the presence of the first differential split command for causing said at least one mass 

parameter in the command indicating that the command storage system to separate said business continuance pair 

is to be executed physically remotely from the local and copy only changed data from the business continuance 

storage system, the second parameter indicating an source volume to a target volume mirror, 

order in which the command is to be executed, and the 19. The method of claim 14, wherein said business 

one or more third parameters indicating one or more continuance operations further comprise the step of using a 

paths via which the command is to be forwarded to the reverse split command for causing said at least one mass 

atleast one mass storage system, the absence of the first storage system to separate said business continuance pair 

parameter from the command indicating that the com- and copy all the contents from a business continuance 

mand is to be executed at the local mass storage system. volume to an associated source volume. 

12. The method of claim 11, wherein said method further 20, The method of claim 14, wherein said business 
comprises the step of using host component software for continuance operations further comprise the step of using a 
processing command syntax to allow a user to specify reverse differential split command for causing said at least 
business continuance operations to be effected on the at least one mass storage system to separate said business continu- 
one mass storage system that is identified by the selected ance pair and copy only changed contents from a business 
identifier. continuance target volume to an associated source volume. 

13. The method of claim 12, wherein said business 

continuance operations comprise the step of using an estab- ***** 
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[57] ABSTRACT 

A multiple virtual space control in a multiple virtual 
storage system having an address translation table used 
to translate a logical address to a real address, a control 
register for holding a start address of the address trans- 
lation table or a space identifier (hereinaAer represented 
by address translation table start address) and an address 
translation buffer containing a pair of logical address 
and real address and an address translation table start 
address for translating a logical address to a real ad- 
dress, in otdcT to upd^ the content of the control 
register to switch the virtual space. A group identifier 
comprising a plurality of bits for identifying an area 
common to a group of virtual spaces is added to an 
entry of the address translation table, an entry of the 
address translation btiffer and the control register. 
When a logical address is to be translated to a real ad- 
dress, if there is an entry having a logical address and an 
address translation table start address equal to the mem- 
ory request logical address and the address translation 
table start address of the control register, or an entry 
having a logical address and a group identifier equal to 
the memory request logical address and the group iden- 
tifier of the control register, in the address translation 
buffer, the real address of the entry is rendered valid 
and used for memory access. 

14 Oainis, 1 Drawing Sheet 
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NfETHOD AND APPARATUS FOR GENERATING 
A REAL ADDRESS MULTIPLE VIRTUAL 
ADDRESS SPACES OF A STORAGE 

BACKGROUND OF THE INVENTION 

The present invention relates to method and appara- 
tus for translating an address of a computer system 
which adopts a virtual memory scheme, and more par- 
ticularly to method and apparatus for controlling multi- 
ple virtual spaces suitable for address translation where 
there is a common area among a portion of the virtual 
address space in a multiple virtual storage system. 

In the computer system of the virtual memory 
scheme, a memory address is given by an address on a 
virtual space (that is, a logical address). Accordingly, 
when a main storage is to be accessed, it is necessary to 
translate the logical address to an address on the main 
storage (that is, a real address). The address translation 
is carried out by looking up an address translation table ^ 
(usually comprising a segment table and page tables) 
provided in the main storage. Pairs of logical addresses 
and real addresses obtained by the address translation 
are usually registered or stored in an address translation 
buffer or translation lookaside buffer (TLB), and for the 
logical address requested for the memory access, the 
corresponding real address is looked up from the TLB 
so that the address translation is carried out at a high 
speed. 

On the other hand, in a multi-process multiple virtual ^0 
storage system, a virtual space is allotted for each job 
and an address translation table is provided for each 
virtual space. Switching of virtual spaces due to switch- 
ing of jobs is carried out under control of an operating 
system (OS) of a computer system, such as Hitachi 35 
VOS3, by rewriting a start address of the address trans- 
lation table or rewriting a content of an address control 
register (ATOR) which holds a space identifier. In this 
case, in addition to the logical address (LA) and the real 
address (RA), the content (A TO) of the address control 40 
register are held in the TLB. When the object real ad- 
dress is to be looked up from the TLB, the coincidence 
of the corresponding virtual spaces as well as the coinci- 
dence of the logical addresses is checked. 

In such a multiple virtual storage system, a common 4S 
area to respective virtual spaces is indicated by adding 
a common segment bit (C bit) in the TLB. When the C 
bit Is **P (indicating the presence of the common area), 
the start address of the address translation table or the 
space identifier held in the TLB is ignored, and if the 50 
logical address from the memory address register and 
the logical address of the TLB are equal, the corre- 
sponding real address is considered vaJid and the ac- 
cessing to the main storage is carried out thereby. In the 
multiple virtual computer system, a computer identifler 35 
(CN bit) is set to an area common to the virtual comput- 
ers, and a pair consbting of a logical address and a real 
address registered in the TLB b rendered valid by a 
logical product (AND) condition of the C bit and the 
CNbit 60 

This type of virtual space control system is shown in 
JP-A-6Q-68443 filed by the assignee of the present appli- 
cation and published on Apr. 19, 1985. 

The prior art described above is effective in enhanc- 
ing a hit rate of the address trsmslation buffer when the 65 
common area which is common to all virtual spaces is 
used, but it does not give consideration to a case where 
there is a common area among a portion of virtual 
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spaces. When the common area is set for the portion of 
virtual spaces, the hit rate of the address translation 
buffer decreases and the number of times of rewriting of 
the buffer increases. Data in the common area can be 
readily read from or updated by any virtual space so 
that security for the data and programs of the respective 
virtual spaces as not suf5ctently assured. Accordingly, 
the prior art system is hardly applicable to an area 
which is to be shared by only specified spaces. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide a 
method and apparatus for controlling address transla- 
tion of multiple virtual spaces of a multi-process ma- 
chine which is suitable when a partial common area is 
shared by a plurality of virtual spaces. 

It is another object of the present invention to pro- 
vide a method and apparatus for controlling address 
translation of multiple virtual spaces which permits 
setting of a common area without resulting in decrease 
of a hit rate of a TLB among specified virtual spaces 
and which can assure security of data and programs. 

In order to achieve the above objects, in accordance 
with the present invention, a group identifier compris- 
ing a plurality of bits for identifying a group of virtual 
spaces having a partial common area is added to each of 
entries of an address translation table, entries of an ad- 
dress translation buffer and a control register which 
holds a start address of the address translation table or a 
space identifier, and when there b an address translation 
buffer entry having a coincidence between a %drtual 
address from a menx>ry address register and the start 
address of the address translation table or the space 
identifier from the address control register, or there b 
an address translation buffer entry having a coincidence 
between the virtual address (LA) from the memory 
address regbter and the group identifier (GID) from the 
address control regbter, the real address of the address 
translation buffer entry b considered valid and the main 
storage b accessed by that real address. 

When address translation b carried out for one virtual 
space under the control of the operating system (OS) of 
the virtual memory scheme, a virtual address and a real 
address as well as a current start address of the address 
translation table or a space identifier and a group identi- 
fier added to the entry of the corresponding address 
translation table are registered into the entry of the 
address translation buffer* The group identifier com- 
prises a plurality of bits so that it can identify a plurality 
of predetermined groups of virtual ^Mices. 

In the address translation in other virtual spaces hav- 
ing the same group identifier, if there b an entry having 
a coincidence between a requested virtual address and 
the virtual address in the address translation buffer and 
a coinckicnce between a group identifier from the ad- 
dress control register and the group identifier in the 
address translation buffer, the address translation b 
carried out by the address translation buffer and no new 
regbtration b made to the address translation buffer. 
Accordingly, rewriting of the buffer b not necessary. In 
the address translation in other virtual spaces having a 
different group identifier, there b no coincidence be- 
tween group identifiers nor the start addresses of the 
address translation tables or space identifiers. Accord- 
ingly» the address translation b carried out by looking 
up the address translation table and the result is regis- 
tered into the address translation buffer. Thus, even if 
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the virtual addresses are equal, different address transla- 
tion tables are looked up. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram showing one embodiment 
of a system configuration of the present invention, and 

FIG. 2 is a diagram showing an example of a memory 
map of ffluldple virtual spaces considered in the present 
invention. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

One embodiment of the present invention is now 
explained with reference to the accompanying draw- 
ings. 

FIG. 2 shows a memory map of multiple virtual 
spaces considered in the present invention. In FIG. X 
each of the virtual spaces 1-8 has an area 100 (wholly 
common area) which is common to all virtual spaces 
l-«, as described in the above-referenced JP-A-60- 
68443. In the present embodiment, the virtual spaces 
1-3 have a partial common area 101 (Group A), the 
virtual spaces 4 and 5 have a partial common area 102 
(Group B), and the virtual spaces 4» 7 and 8 have a 
partial conmKHi area 103 (Group C). Hereinafter, an 
identifier for the wholly common area 100 is called a 
common segment bit (C), and identifiers for the Groups 
A, B and C (101-103) are called group ID's (GID's). 
The GID may comprises eight bits with respective bits 
corresponding to the spaces 1-8. For the Group A, the 
bits 1-3 are "l'*. for the Group B, the bits 4 and 5 are 
"l**, and for the Group C, the bits 4, 7 and 8 are "*V\ 
The Groups A, B and C may be identified by codes. In 
this case, GID may comprise less than eight bits. 

FIG. 1 shows a block diagram of one embodiment of 35 
an address translation control system of the present 
invention. It shows a hit detector of an address transla- 
tion buffer. 

In FIG. 1, a memory address register (MAR) 1 holds 
a logical address (LA) of a memory access request. An 40 
address translation table start address register (ATOR) 
2 holds a space identifier or an address translation table 
start address (ATO) and a group ID (GID) in order to 
identify a current virtual space. When a virtual space is 
selected and switched under control of an OS, the space 45 
identifier or ATO in the ATOR 2 is updated. In the 
following description, cither the space identifier or the 
ATO is used in the same manner. 

Each entry of the address translation table 3 com- 
prises a real address (RA), a common segment bit (C) 50 
and a group ID (GID). The address translation table 3 
IS provided in a main storage for each virtual space» and 
a start address thereof is designated by the ATO in the 
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the memory address register 1, the ATO (or space iden- 
tifier) of the ATOR 2, and RA, C and GID of the ad- 
dress translation table 3 are registered into the corre- 
sponding entry of the TLB 4. 

An exclusive OR circuit (EX-OR) 5 compares the 
ATO of the ATOR 2 and the ATO of the TLB 4 and 
produces a logical "1** output when they are equal. An 
EX-OR 6 compares the logical address of the memory 
address register 1 and the LA of the TLB 4 and pro- 
duces a logical "1** output when they are equal. An 
EX-OR 7 compares the GID of the ATOR 2 and the 
GID of the TLB 4 and produces a logical **r' output 
when they are equal. Those EX-OR*s produce logical 
**0" signals when compared inputs are not equal. An OR 
circuit 8 constitutes detection means for detecting if a 
virtual space designated by the OS does not belong to 
any common group and produces a logical **V* output 
when at least one of the bits of the GID of the ATOR 
2 is a significant bit and an input logical signal is **V\ An 
AND circuit 9 produces a logical output when the 
outputs of both the EX-OR 7 and the OR circuit 8 are 
**V\ An OR circuit 10 produces a logical **V* output 
when one of the output of the EX-OR 5. the C bit of the 
TLB 4 and the output of the AND circuit 9 is **V\ An 
AND circuit 11 produces a logical "I** output when the 
outputs of both of the EX-OR 6 and the OR circuit 10 
are **r*. The output of the AND circuit II is an address 
translation buffer hit signal HIT. When the signal HIT 
is **V\ the RA of the TLB 4 is rendered valid and the 
memory is accessed by that real address. 

The operation of FIG. 1 is explained with reference 
to the memory nuip shown in FIG. 2. 

It is assumed that values of the area 101 of the Group 
A of the virtual space 1 have been registered in the 
ATO. LA, RA, C and GID of the entries of the TLB 4» 
and the ATO and GID of the virtual space 2 have been 
set in the ATOR 2. If the logical address of the memory 
address register MAR 1 designates an area of the Group 
A, the EX-OR 6 produces a logical **1" output. Since 
the ATO of the ATOR 2 and the ATO of the TLB 4 are 
now not equal* the EX-OR prtxluces a logical **0" out- 
put. Since the area 101 of the Group A is not the wholly 
common area 100, the C bit of the TLB is *'0". In this 
case, if the group ID (OlD) has not been set, the OR 
circuit 10 produces a logical "C* output because the 
circuits 7-9 produce logi^ '"O** outputs and the AND 
circuit 11 produces a logical ••(>" output which indicates 
that the correspondmg logical address does not exist in 
the TLB (not in TLB). 

On the other hand, if values indicating the common 
Group A to the virtual spaces 1 and 2 (that is, bits 1-3 
of the GID are have been set in the GID of the 
ATOR 2, the outputs of both the EX-OR 7 and the OR 
8 are "1** and the OR circuit 10 produces a logical 'T* 



ATOR 2. The address translation table usually com- . - . , . 

prises a segment table and page tables, but Uiey are 55 ou^ut. As a result, the HIT signal of the AND curcmt 
shown as one table in the present embodiment for the 11 is indicating that a real addre^ corr<»ponduig to 
purpose of simplification. 

Each entry of the address translation buffer (TLB) 4 
comprises an address translation table start address 
(ATO), a logical address (LA), a real address (RA)» a 60 
common segment bit (C) and a group ID (GID). While 



only one entry is shown, the TLB 4 actually has a plu- 
rality of entries and a desired entry is accessed by a 
predetermined bit such as a start address of the memory 
address register (MAR) 1. This method is referred to as 
a set associative address translation buffer control. 
When the address translation is carried out by using the 
address translation table 3, the logical address (LA) of 
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the logica] address esusts in the TLB (in TLB). Thus, 
the main memory is acc^essed by using the RA of the 
TLB 4 as the real address. It is noted that the space 4 
included in Group B and Group C is relatively small in 
the numt>er of reference thereto. 

In accordance with the present invention, the address 
translation can be attained by the same address transla- 
tion buffer entry for the partially common area in difTer- 
ent virtual spaces. Thus, in the set associative address 
translation buffer control, the number of times of updat- 
ing of the address translation buffer decreases and the 
buffer hit rate is enhanced. Further, the accessing to«a 
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common area of a group of virtual spaces by a virtual 
space of another group can be inhibited by setting dif- 
ferent group ID'S for the respective groups. 
We claim: 

1. A method of generating a real address in multiple 
virtual address spaces of a storage having multiple vir- 
tual storage spaces^ an address translation table for each 
virtual address space to translate a logical address to a 
real address, an address control register for storing a 
start address of the address translation table or a virtual 
address space identifier, and an address translation 
buffer c^Mble of storing copies of entries contained in 
the address translation table, said method comprising 
the steps of: 

providing group identifiers in each of the entries in 
said address translation table, said address transla- 
tion buffer and the entry in said address control 
register for identifying a respective area shared by 
a specific group of virtual spaces; 

determining whether there is at least one entry in the 20 
address translation buffer whose logical address 
and address translation table start address conform 
with the logical address supplied for a memory 
request and the address translation table start ad- 
dress in said address control register, or whether an 23 
entry exists in the address translation buffer whose 
logical address and group identifier conform to the 
logical address supplied for memory access and the 
group identifier of said address control register; 
and 30 

if there is at least one conformity, identifying the real 
address of said entry as a valid real address for 
memory access. 

2. A method according to claim 1 wherein the shared 
areas include a plurality of different areas including a 35 
predetermined combination of portions of virtual ad- 
dress spaces smaller in number than the total number of 
virtual spaces, each of said different areas having a 
group identifier for identifying the particular area, said 
areas being independent from one another and non- 40 
accessible by each other* 

3. A method according to claim 1 wherein the shared 
areas include an area common to at least one first and 
one second virtual address space, each virtual address 
space having a plurality of limited common sub-areas, 45 
and data in the common sub-areas being managed by an 
operating system. 

4. A method according to claim 3 wherein said first 
virtiial address space has a higher priority than said 
second virtual address space. 

5. A method according to claim 1 wherein said deter- 
mining step includes sub-steps of (1) comparing the 
logical address supplied from a memory address register 
with the logical address in said address translation 
buffer to produce a first coincidence signal when they 
are equal, and comparing the group identifier of said 
address control register and the group identifier of said 
address translation buffer to produce a second coinci- 
dence signal when they are equal, and (2) determining 



7. A method according to claim 1 further comprising 
a step of copying the entry in the address translation 
table into said address tran^tion buffer if it is found 
that the corresponding entry does not exist in the ad- 
dress translaticH) buffer. 

8. A virtual address space control apparatus in a mul- 
tiple virtual storage system, comprising: 

(a) an address translation table having entries each 
including a pair of logical and real addresses and a 
group identifier for identifying storage areas com- 
m<m to specified groups of virtual address spaces, 
for translating a logical address to a real address 
under control of an operating system; 

(b) an address translation buffer having logical and 
real address pairs copied from the address transla- 
tion table, and an associated group identifier and an 
associated virtual address space identifier for trans- 
lating a logical address to a real address; 

(c) an address control register specifying a virtual 
address space identifier and a group identifier; and 

(d) detection means responsive to supply of a logical 
address for memory access for detecting the pres- 
ence of entries in the address translation buffer of 
the kind whose logical address and virtual address 
space identifier are equal to the logical address 
supplied for memory access and the virtual address 
space identifier of said address control register, or 
for detecting entries of the kind whose logical ad- 
dress and group identifier are equal to the logical 
address supplied for memory access and the group 
identifier of said address control register; 

said detection means, when the presence of either of 
said entries is detected, producing a signal for ren- 
dering the real address of said address translation 
buffer valid as a memory accessing address. 

9. A control apparatus according to claim 8 wherein 
said group identifier includes a plurality of bits, while a 
single bit identifying a common area to all virtual ad- 
dress spaces is provided in said address translation table 
and in the address translation buffer. 

10. A control apparatus according to claim 9 wherein 
said detection means include means for comparing the 
logical address supplied for a memory access from a 
memory address register and the content of the corre- 
sponding entry of said address translation buffer to 
produce a first coincidence signal when they are equal, 
means for comparing the group identifier of said ad- 
dress control register and the content of the corre- 
sponding entry of said address translation buffer to 

50 produce a second coincidence signal when they are 
equal, and select signal producing means responsive to 
simultaneous occurrence of said first coincidence signal 
and said second coincidence signal for producing a 
signal to select the real address of the corresponding 
entry in said address translation buffer as an address for 
memory access. 

11. A control apparatus according to claim 10 
wherein said means for producing the second coinci- 
dence signal includes means for inhibiting the second 
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that the corresponding entry in said address translation 60 coincidence signal when it is detected that the group 



buffer is valid when said first coincidence signal and 
said second coincidence signal are simultaneously pres- 
ent. 

6. A method according to claim 5 wherein said sec- 
ond coinciding signal is inhibited when it is detected 65 
that the group identifier from said address control regis- 
ter does not belong to any group of virtual address 
spaces with shared areas. 



identifier from said address control register does not 
belong to any group of virtual address spaces having 
common areas. 

12. A control apparatus according to claim 10 
wherein said detection means further includes means for 
comparing the control of the output from said address 
control register and the content of the corresponding 
entry of said address translation buffer to produce a 
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third coincidence signal when they arc equal, said select 
signal producing means producing a select signal upon 
the simultaneous occurrence of said first coincidence 
signal and said third coincidence signal. 

13. A control apparatus according to claim 12 
wherein said means for producing a select signal re- 
sponds to simultaneous occurrence of a signal indicating 
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the common area to all virtual address spaces in said 
address translation buffer and said first coincidence 
signal. 

14. A control apparatus according to claim 8 wherem 
5 said virtual address space identifier comprises a start 
address of said address translation table. 
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